Oct 31 03:04:01.882238 ip-10-0-139-164 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 31 03:04:01.882252 ip-10-0-139-164 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 31 03:04:01.882261 ip-10-0-139-164 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 31 03:04:01.882575 ip-10-0-139-164 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 31 03:04:13.135781 ip-10-0-139-164 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 31 03:04:13.135795 ip-10-0-139-164 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4b3ec0f4ec1541f5baba760f45e2bfa3 --
Oct 31 03:06:22.816307 ip-10-0-139-164 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 03:06:23.275099 ip-10-0-139-164 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:06:23.275099 ip-10-0-139-164 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 03:06:23.275099 ip-10-0-139-164 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:06:23.275099 ip-10-0-139-164 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:06:23.278714 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278568    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:23.278714 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278713    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:23.278714 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278718    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:23.278714 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278721    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278725    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278728    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278731    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278734    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278736    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278739    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278742    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278745    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278747    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278750    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278752    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278761    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278764    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278767    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278770    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278772    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278775    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278777    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278780    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:23.278849 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278782    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278785    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278790    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278792    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278795    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278798    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278800    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278803    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278805    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278808    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278811    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278813    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278816    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278818    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278822    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278825    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278827    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278830    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278834    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:23.279337 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278837    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278840    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278843    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278845    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278848    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278851    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278853    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278856    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278859    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278862    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278864    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278867    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278870    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278872    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278888    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278891    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278896    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278901    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278906    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:23.279812 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278911    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278914    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278917    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278920    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278923    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278925    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278928    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278931    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278934    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278936    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278940    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278944    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278947    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278950    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278953    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278956    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278959    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278962    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278964    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:23.280285 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278967    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278970    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278972    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278975    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278978    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278980    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278983    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278985    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278988    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278991    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278994    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278996    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.278999    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279002    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279005    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279008    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279010    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279013    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279015    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279018    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:23.280773 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279020    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279024    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279026    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279029    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279033    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279036    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279905    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279914    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279918    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279920    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279923    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279926    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279929    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279932    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279934    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279937    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279940    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279943    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279945    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:23.281267 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279948    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279950    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279954    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279956    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279959    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279962    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279965    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279967    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279970    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279973    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279975    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279978    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279980    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279983    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279987    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279989    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279992    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279995    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.279998    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280002    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:23.281723 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280005    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280007    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280010    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280012    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280016    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280020    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280024    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280027    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280031    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280034    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280037    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280040    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280043    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280045    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280048    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280050    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280053    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280056    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280059    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:23.282237 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280062    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280064    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280067    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280070    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280073    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280075    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280078    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280083    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280086    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280088    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280091    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280094    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280096    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280099    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280102    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280105    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280107    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280110    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280112    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:23.282712 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280115    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280118    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280121    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280125    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280129    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280132    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280136    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280138    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280141    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280143    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280146    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280148    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280151    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280154    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280156    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280159    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280162    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280164    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280167    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:23.283281 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280170    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280172    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280175    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280178    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280180    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280183    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280185    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280187    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280190    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280193    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280195    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280198    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280201    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280203    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280206    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.280210    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282341    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282350    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282353    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282356    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:23.283737 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282359    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282362    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282364    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282367    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282371    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282373    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282376    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282379    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282382    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282385    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282388    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282390    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282393    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282396    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282398    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282401    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282404    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282407    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282409    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282412    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:23.284233 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282416    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282419    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282421    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282424    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282427    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282430    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282432    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282435    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282437    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282441    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282444    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282447    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282450    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282453    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282456    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282459    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282461    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282464    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282466    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:23.284727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282469    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282472    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282475    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282478    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282480    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282483    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282486    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282488    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282491    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282494    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282496    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282499    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282501    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282506    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282510    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282513    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282516    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282518    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282522    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282524    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:23.285186 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282527    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282530    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282532    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282535    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282539    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282542    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282544    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282547    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282550    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282552    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282555    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282558    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282560    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282563    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282566    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282568    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282571    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282574    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282576    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:23.285653 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282579    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282582    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282584    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282588    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282590    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282593    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282595    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282598    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282601    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282603    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282606    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282609    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282612    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282614    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282617    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282619    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282622    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282624    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282627    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282629    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:23.286218 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282631    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282634    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282636    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.282639    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285010    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285024    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285033    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285038    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285044    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285047    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285052    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285057    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285061    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285064    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285068    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285072    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285075    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285078    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285082    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285085    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285088    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285092    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285096    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285099    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 03:06:23.286684 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285102    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285105    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285109    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285117    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285120    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285124    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285127    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285131    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285134    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285137    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285140    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285143    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285148    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285151    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285154    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285157    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285161    2576 flags.go:64] FLAG: --enable-server="true"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285164    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285169    2576 flags.go:64] FLAG: --event-burst="100"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285172    2576 flags.go:64] FLAG: --event-qps="50"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285176    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285179    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285182    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285188    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 03:06:23.287253 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285192    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285195    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285199    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285202    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285206    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285209    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285213    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285216    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285219    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285222    2576 flags.go:64] FLAG: --feature-gates=""
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285227    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285230    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285234    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285237    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285241    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285244    2576 flags.go:64] FLAG: --help="false"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285247    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285250    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285253    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285256    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285260    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285263    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285267    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 03:06:23.287888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285271    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285274    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285277    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285280    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285283    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285286    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285289    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285292    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285296    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285299    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285302    2576 flags.go:64] FLAG: --lock-file=""
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285305    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285308    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285311    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285318    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285322    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285325    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285329    2576 flags.go:64] FLAG: --logging-format="text"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285332    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285336    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285339    2576 flags.go:64] FLAG: --manifest-url=""
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285342    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285347    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285351    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285355    2576 flags.go:64] FLAG: --max-pods="110"
Oct 31 03:06:23.288452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285358    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285361    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285364    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285367    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285370    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285374    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285377    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285386    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285389    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285393    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285396    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285399    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285402    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285405    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285408    2576 flags.go:64] FLAG: --port="10250"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285412    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285415    2576 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0a978aae77f6894a2"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285418    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285422    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285425    2576 flags.go:64] FLAG: --register-node="true"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285428    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285432    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285435    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285438    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285442    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285446    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 03:06:23.289080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285450    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285453    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285457    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285459    2576 flags.go:64] FLAG: --runonce="false"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285462    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285466    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285469    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285472    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285475    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285478    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285482    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285485    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285488    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285492    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285495    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285498    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285501    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285504    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285507    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285513    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285516    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285519    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285522    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285525    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285528    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 03:06:23.289688 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285531    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285534    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285537    2576 flags.go:64] FLAG: --v="2"
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285542    2576 flags.go:64] FLAG: --version="false"
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285547    2576 flags.go:64] FLAG: --vmodule=""
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285552    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.285556    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285716    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285721    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285725    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285728    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285730    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285733    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285736    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285738    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285741    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285744    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285746    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285749    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285752    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:23.290981 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285754    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285756    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285759    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285762    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285764    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285767    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285770    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285773    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285776    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285778    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285780    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285784    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285788    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285791    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285794    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285796    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285799    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285801    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285804    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:23.291462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285808    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285810    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285813    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285816    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285818    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285821    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285824    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285826    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285828    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285831    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285834    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285836    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285839    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285842    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285844    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285847    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285849    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285853    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285857    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:23.291993 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285860    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285862    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285865    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285867    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285870    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285887    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285890    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285892    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285895    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285898    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285901    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285903    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285906    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285908    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285912    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285915    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285918    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285920    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285923    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285925    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:23.292466 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285931    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285934    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285936    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285939    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285942    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285945    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285948    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285950    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285952    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285955    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285957    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285960    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285963    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285965    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285968    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285970    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285973    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285975    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285978    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285980    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:23.292957 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285983    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285985    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285988    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285991    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285993    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285996    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.285998    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286001    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286006    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286008    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286011    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286014    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286017    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286020    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286023    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.286061    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:06:23.293417 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286108    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286114    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286117    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286120    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286123    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286126    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286129    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286132    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286134    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286137    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286139    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286142    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286145    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286147    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286150    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286153    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286155    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286158    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286160    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:23.293797 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286163    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286166    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286169    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286171    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286174    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286179    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286182    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286185    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286189    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286193    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286197    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286200    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286203    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286206    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286208    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286211    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286214    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286216    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286219    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:23.294269 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286223    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286226    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286228    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286231    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286234    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286236    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286239    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286241    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286244    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286247    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286249    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286252    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286254    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286257    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286261    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286264    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286267    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286271    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286275    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:23.294732 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286279    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286282    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286284    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286287    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286290    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286293    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286296    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286299    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286301    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286304    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286306    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286309    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286312    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286314    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286317    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286320    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286322    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286325    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286327    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286330    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:23.295264 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286332    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286335    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286337    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286340    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286342    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286345    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286347    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286350    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286352    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286355    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286357    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286362    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286364    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286367    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286370    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286372    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286376    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286379    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286381    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286384    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:23.295734 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286387    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286389    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286392    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286395    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286397    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286400    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286402    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286406    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.286408    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.294404    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.294423    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.295200    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.295206    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295303    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295310    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295313    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295316    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295319    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295323    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295326    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:23.296252 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295329    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295332    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295335    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295338    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295341    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295349    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295353    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295355    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295358    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295361    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295365    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295369    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295372    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295375    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295378    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295381    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295383    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295387    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295392    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:23.296727 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295395    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295398    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295401    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295404    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295407    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295410    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295413    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295415    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295418    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295421    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295423    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295426    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295428    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295431    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295434    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295437    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295439    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295442    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295445    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:23.297231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295453    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295456    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295459    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295461    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295464    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295467    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295469    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295472    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295474    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295477    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295480    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295482    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295485    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295488    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295491    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295493    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295496    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295499    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295502    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295504    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:23.297686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295507    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295510    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295512    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295515    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295518    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295521    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295523    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295525    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295528    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295531    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295533    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295536    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295538    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295547    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295550    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295552    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295555    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295558    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295562    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295564    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:23.298193 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295567    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295570    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295572    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295575    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295577    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295584    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295587    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295590    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295592    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295595    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295598    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295600    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295603    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295606    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295608    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295611    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295613    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295616    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295619    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295621    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:23.298655 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295624    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.295659    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295922    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295929    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295932    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295935    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295943    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295946    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295955    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295957    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295960    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295963    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295966    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295969    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295972    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295974    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:23.299231 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295977    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295979    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295982    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295985    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295988    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295991    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295994    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295997    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.295999    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296002    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296005    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296007    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296010    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296013    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296016    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296019    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296021    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296024    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296026    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296029    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:23.299604 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296032    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296035    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296037    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296046    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296049    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296051    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296055    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296057    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296060    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296062    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296065    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296068    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296072    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296074    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296077    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296079    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296082    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296085    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296087    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296090    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:23.300085 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296092    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296095    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296097    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296100    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296102    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296105    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296107    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296110    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296113    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296115    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296118    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296120    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296123    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296125    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296128    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296130    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296138    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296141    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296144    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:23.300698 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296146    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296149    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296151    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296154    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296157    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296159    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296162    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296172    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296175    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296177    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296180    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296182    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296185    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296188    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296190    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296193    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296195    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296198    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296200    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:23.301230 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296203    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296205    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296208    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296210    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296213    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296216    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296218    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296221    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296223    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296228    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296231    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296239    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296242    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296246    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.296272    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:06:23.301686 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296322    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296326    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296329    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296331    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296334    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296338    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296340    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296343    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296345    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296348    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296350    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296353    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296356    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296359    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296361    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296364    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296367    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296369    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296372    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:23.302081 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296374    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296377    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296379    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296382    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296384    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296388    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296392    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296395    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296397    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296401    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296411    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296414    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296417    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296419    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296422    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296424    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296427    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296430    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296432    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:23.302533 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296435    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296438    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296441    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296443    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296446    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296451    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296454    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296457    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296460    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296462    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296466    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296468    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296471    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296473    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296476    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296479    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296481    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296483    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296486    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296489    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:23.303003 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296491    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296494    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296497    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296500    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296508    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296511    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296513    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296516    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296518    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296521    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296524    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296527    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296529    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296532    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296534    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296537    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296540    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296544    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296547    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296549    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:23.303462 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296551    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296555    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296558    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296560    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296563    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296565    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296568    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296570    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296573    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296575    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296578    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296580    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296583    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296585    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296588    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296590    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296593    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296602    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296605    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:23.303961 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296608    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296610    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296613    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296616    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296618    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296621    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296623    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296626    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:23.296629    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.296689    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.299147    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.301634    2576 server.go:1014] "Starting client certificate rotation"
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.301734    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.301765    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:06:23.304450 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.303772    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:06:23.308808 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.308780    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:06:23.320791 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.320767    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 31 03:06:23.325096 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.325075    2576 log.go:25] "Validated CRI v1 image API"
Oct 31 03:06:23.326128 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.326089    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 03:06:23.331010 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.330986    2576 fs.go:135] Filesystem UUIDs: map[0a526239-b7b9-4296-8a4e-9c591927164b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fedca4f8-ac5a-4b7d-ad03-1fa557183428:/dev/nvme0n1p4]
Oct 31 03:06:23.331090 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.331009    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 03:06:23.337868 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.337723    2576 manager.go:217] Machine: {Timestamp:2026-10-31 03:06:23.334945093 +0000 UTC m=+0.382907272 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100794 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c17d45e05717897c12e1aed72c632 SystemUUID:ec2c17d4-5e05-7178-97c1-2e1aed72c632 BootID:4b3ec0f4-ec15-41f5-baba-760f45e2bfa3 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:cf:9d:b4:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:cf:9d:b4:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:e2:6b:9d:37:e0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 03:06:23.337868 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.337857    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 03:06:23.338063 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.338048    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 03:06:23.341078 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.341038    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 03:06:23.341243 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.341081    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-164.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 03:06:23.341292 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.341247    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 03:06:23.341292 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.341255    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 03:06:23.341292 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.341283    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:06:23.341371 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.341296    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:06:23.342320 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.342307    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 03:06:23.345923 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.345909    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 03:06:23.346091 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.346080    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 03:06:23.346924 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.346913    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 03:06:23.346972 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.346936    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 03:06:23.346972 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.346952    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 03:06:23.346972 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.346962    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 31 03:06:23.346972 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.346972    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 03:06:23.347420 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.347404    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:06:23.349985 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.349969    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 03:06:23.356186 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.356160    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:06:23.357437 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.357415    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 03:06:23.357515 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.357453    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 03:06:23.357721 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.357706    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 03:06:23.357774 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.357730    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 03:06:23.357774 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.357738    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 03:06:23.357774 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.357744    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 03:06:23.357774 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.357751    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 03:06:23.357774 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.357757    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 03:06:23.359385 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.359374    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 03:06:23.359385 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.359386    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 03:06:23.359450 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.359394    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 03:06:23.359450 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.359401    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 03:06:23.359450 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.359411    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 03:06:23.359450 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.359422    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 03:06:23.360655 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.360645    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 03:06:23.360655 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.360656    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 03:06:23.364740 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.364725    2576 server.go:1263] "Started kubelet"
Oct 31 03:06:23.364871 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.364812    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 03:06:23.364966 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.364909    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 03:06:23.365016 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.364987    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 03:06:23.365654 ip-10-0-139-164 systemd[1]: Started Kubernetes Kubelet.
Oct 31 03:06:23.366243 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.366219    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 03:06:23.367085 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.367071    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 03:06:23.370175 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.370157    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 03:06:23.372205 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.372179    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 03:06:23.372803 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.372783    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 03:06:23.373025 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.373000    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:06:23.373687 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.373668    2576 factory.go:55] Registering systemd factory
Oct 31 03:06:23.373687 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.373684    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 03:06:23.373813 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.373700    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 03:06:23.373813 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.373683    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 03:06:23.373813 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.373687    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 31 03:06:23.373813 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.373810    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 03:06:23.373994 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.373818    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 03:06:23.374060 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.374046    2576 factory.go:153] Registering CRI-O factory
Oct 31 03:06:23.374107 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.374066    2576 factory.go:223] Registration of the crio container factory successfully
Oct 31 03:06:23.374153 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.374123    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 03:06:23.374153 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.374142    2576 factory.go:103] Registering Raw factory
Oct 31 03:06:23.374244 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.374158    2576 manager.go:1199] Started watching for new ooms in manager
Oct 31 03:06:23.375112 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.375098    2576 manager.go:319] Starting recovery of all containers
Oct 31 03:06:23.385186 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.385158    2576 manager.go:324] Recovery completed
Oct 31 03:06:23.391162 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.391134    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:23.393418 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.393392    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:23.393511 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.393434    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:23.393511 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.393445    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:23.394053 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.394033    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 31 03:06:23.394119 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.394052    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 03:06:23.394119 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.394056    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:23.394119 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.394073    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 03:06:23.395547 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.395529    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:23.395547 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.395548    2576 policy_none.go:50] "Start"
Oct 31 03:06:23.395648 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.395562    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 03:06:23.395648 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.395564    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:23.395648 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.395577    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 03:06:23.395648 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.395581    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:23.396832 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.396818    2576 policy_none.go:44] "Start"
Oct 31 03:06:23.434514 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.434280    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 03:06:23.454399 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.454367    2576 manager.go:341] "Starting Device Plugin manager"
Oct 31 03:06:23.454546 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.454422    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 03:06:23.454546 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.454433    2576 server.go:87] "Starting device plugin registration server"
Oct 31 03:06:23.454722 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.454709    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 03:06:23.454768 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.454727    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 03:06:23.454835 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.454821    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 03:06:23.454932 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.454920    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 03:06:23.454990 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.454933    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 03:06:23.455649 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.455622    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 03:06:23.455747 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.455668    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:06:23.494084 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.494050    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-164.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 03:06:23.494084 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.494074    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-164.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 03:06:23.495372 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.494060    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-164.us-west-2.compute.internal.18e37e55bef5d884  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-164.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-164.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 03:06:23.3646921 +0000 UTC m=+0.412654279,LastTimestamp:2026-10-31 03:06:23.3646921 +0000 UTC m=+0.412654279,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-164.us-west-2.compute.internal,}"
Oct 31 03:06:23.510506 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.510460    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 03:06:23.511834 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.511803    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 03:06:23.511969 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.511838    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 03:06:23.511969 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.511863    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 03:06:23.511969 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.511928    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 03:06:23.555094 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.554997    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:23.556929 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.556859    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 03:06:23.557086 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.556947    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h9cwz"
Oct 31 03:06:23.557086 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.556990    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:23.557086 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.557023    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:23.557086 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.557040    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:23.557086 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.557074    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.557260 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.557172    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-164.us-west-2.compute.internal.18e37e55c0ac2e58  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-164.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-164.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-164.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 03:06:23.39341884 +0000 UTC m=+0.441381021,LastTimestamp:2026-10-31 03:06:23.39341884 +0000 UTC m=+0.441381021,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-164.us-west-2.compute.internal,}"
Oct 31 03:06:23.557661 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.557635    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 03:06:23.557661 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.557639    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-164.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 03:06:23.557783 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.557637    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 03:06:23.612976 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.612936    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"]
Oct 31 03:06:23.613167 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.613013    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:23.614147 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.614130    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:23.614210 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.614162    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:23.614210 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.614172    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:23.615651 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.615638    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:23.615832 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.615805    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.615921 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.615846    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:23.616506 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.616491    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:23.616591 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.616521    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:23.616591 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.616536    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:23.616591 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.616492    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:23.616732 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.616593    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:23.616732 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.616603    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:23.617945 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.617926    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.618045 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.617957    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:23.619190 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.619174    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:23.619278 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.619202    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:23.619278 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.619215    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:23.640646 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.640618    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:23.641237 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.641216    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found" node="ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.645482 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.645446    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found" node="ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.646014 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.645994    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.646117 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.646017    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-164.us-west-2.compute.internal\": node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:06:23.675795 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.675756    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41bbbd7b4d4d448be0c950dd63a0f8ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"41bbbd7b4d4d448be0c950dd63a0f8ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.675975 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.675842    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41bbbd7b4d4d448be0c950dd63a0f8ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"41bbbd7b4d4d448be0c950dd63a0f8ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.675975 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.675903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d510a50aad744fb4754386c4136c4a71-config\") pod \"kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"d510a50aad744fb4754386c4136c4a71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.759271 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.759236    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:06:23.776627 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.776601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41bbbd7b4d4d448be0c950dd63a0f8ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"41bbbd7b4d4d448be0c950dd63a0f8ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.776700 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.776631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d510a50aad744fb4754386c4136c4a71-config\") pod \"kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"d510a50aad744fb4754386c4136c4a71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.776700 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.776653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41bbbd7b4d4d448be0c950dd63a0f8ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"41bbbd7b4d4d448be0c950dd63a0f8ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.776773 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.776719    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d510a50aad744fb4754386c4136c4a71-config\") pod \"kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"d510a50aad744fb4754386c4136c4a71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.776773 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.776725    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41bbbd7b4d4d448be0c950dd63a0f8ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"41bbbd7b4d4d448be0c950dd63a0f8ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.776849 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.776789    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41bbbd7b4d4d448be0c950dd63a0f8ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"41bbbd7b4d4d448be0c950dd63a0f8ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.859986 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.859912    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:06:23.945472 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.945426    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.950772 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:23.949912    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:23.960357 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:23.960329    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:06:24.061794 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:24.061753    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:06:24.162822 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:24.162706    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:06:24.263530 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:24.263497    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:06:24.301955 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:24.301918    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 03:06:24.302456 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:24.302116    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 03:06:24.302456 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:24.302155    2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a9b46fe1a16de407895db29e9b878039-acea73c2d7879f85.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-164.us-west-2.compute.internal.18e37e55c0ac7dd9\": read tcp 10.0.139.164:36274->3.139.21.200:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-164.us-west-2.compute.internal.18e37e55c0ac7dd9  default   5402 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-164.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-164.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-164.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 03:06:23 +0000 UTC,LastTimestamp:2026-10-31 03:06:23.614167319 +0000 UTC m=+0.662129497,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-164.us-west-2.compute.internal,}"
Oct 31 03:06:24.364660 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:24.364615    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:06:24.465938 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:24.465841    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:06:24.562342 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:24.562274    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:01:24 +0000 UTC" deadline="2028-06-30 12:01:17.572785523 +0000 UTC"
Oct 31 03:06:24.562342 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:24.562329    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14600h54m53.010459609s"
Oct 31 03:06:24.566794 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:24.566772    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:06:24.622113 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:24.622079    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:24.672770 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:24.672745    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:24.673851 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:24.673833    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:24.675290 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:24.675272    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:24.733592 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:24.733524    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:06:24.734726 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:24.734708    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:06:24.794150 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:24.794117    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:06:25.298989 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:25.298926    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd510a50aad744fb4754386c4136c4a71.slice/crio-8dda0ac8387681a083e0ccbaa2c30e7472baba88429468c1ce6d7bde63099e5a WatchSource:0}: Error finding container 8dda0ac8387681a083e0ccbaa2c30e7472baba88429468c1ce6d7bde63099e5a: Status 404 returned error can't find the container with id 8dda0ac8387681a083e0ccbaa2c30e7472baba88429468c1ce6d7bde63099e5a
Oct 31 03:06:25.299384 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:25.299361    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41bbbd7b4d4d448be0c950dd63a0f8ce.slice/crio-482d8994a27be3a92c38394df337d8d0450d366ebd3bd3a47535b14a4d4be0da WatchSource:0}: Error finding container 482d8994a27be3a92c38394df337d8d0450d366ebd3bd3a47535b14a4d4be0da: Status 404 returned error can't find the container with id 482d8994a27be3a92c38394df337d8d0450d366ebd3bd3a47535b14a4d4be0da
Oct 31 03:06:25.308212 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.308181    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:06:25.349575 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.349532    2576 apiserver.go:52] "Watching apiserver"
Oct 31 03:06:25.373481 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.373452    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 03:06:25.502894 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.502815    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 03:06:25.503176 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.503157    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-mr7hm","openshift-ovn-kubernetes/ovnkube-node-nv7ws","kube-system/global-pull-secret-syncer-ldcbk","openshift-multus/network-metrics-daemon-sl5hn","openshift-network-operator/iptables-alerter-dkpt8","kube-system/konnectivity-agent-x4wzm","openshift-multus/multus-68wpw","openshift-multus/multus-additional-cni-plugins-dh7cw","kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj","openshift-dns/node-resolver-fbw9q","openshift-cluster-node-tuning-operator/tuned-9775g","openshift-image-registry/node-ca-bzwqk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"]
Oct 31 03:06:25.505960 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.505932    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.507434 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.507409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:06:25.509086 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.509059    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:25.509193 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:25.509150    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:25.510447 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.510425    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:25.510531 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:25.510509    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:25.512039 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.512020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:06:25.513447 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.513423    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:06:25.515092 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.515069    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.516547 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.516527    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:25.516654 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:25.516598    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:25.518759 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.518732    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.520370 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.520349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.522502 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.522483    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:06:25.523722 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.523670    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal" event={"ID":"d510a50aad744fb4754386c4136c4a71","Type":"ContainerStarted","Data":"8dda0ac8387681a083e0ccbaa2c30e7472baba88429468c1ce6d7bde63099e5a"}
Oct 31 03:06:25.523820 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.523730    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.526792 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.526763    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal" event={"ID":"41bbbd7b4d4d448be0c950dd63a0f8ce","Type":"ContainerStarted","Data":"482d8994a27be3a92c38394df337d8d0450d366ebd3bd3a47535b14a4d4be0da"}
Oct 31 03:06:25.559066 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.558971    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 03:06:25.559066 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.558979    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 03:06:25.559066 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.559006    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 03:06:25.559066 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.559005    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 03:06:25.559066 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.559031    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8l9rs\""
Oct 31 03:06:25.559428 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.559318    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 03:06:25.559542 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.559524    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 03:06:25.559599 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.559561    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 03:06:25.559599 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.559594    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lsfgs\""
Oct 31 03:06:25.564041 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.564017    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:06:25.574805 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.574771    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 03:06:25.587859 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.587810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysctl-conf\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.588025 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.587869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d35c8150-ce21-4660-ad5d-e5cc1fce684b-kubelet-config\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:25.588025 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.587921    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-run-netns\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.588025 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.587957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnpf\" (UniqueName: \"kubernetes.io/projected/0fbec67c-68ae-48f9-b780-bff7931da269-kube-api-access-4dnpf\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:25.588234 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-registration-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.588234 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-kubernetes\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.588234 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-systemd\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.588234 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588142    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-k8s-cni-cncf-io\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.588234 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-conf-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.588234 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-daemon-config\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.588234 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588225    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-kubelet\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.588521 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588256    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-cni-bin\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.588521 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:25.588521 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d78tf\" (UniqueName: \"kubernetes.io/projected/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-kube-api-access-d78tf\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:06:25.588521 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.588521 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588383    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-system-cni-dir\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.588521 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.588521 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.588521 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588507    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-device-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.588909 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-tmp-dir\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:06:25.588909 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588571    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-cnibin\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.588909 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/382f0f50-a8e4-463d-a64c-710cfffdc963-cni-binary-copy\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.588909 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588605    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-netns\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.588909 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588624    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tknn7\" (UniqueName: \"kubernetes.io/projected/382f0f50-a8e4-463d-a64c-710cfffdc963-kube-api-access-tknn7\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.588909 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-iptables-alerter-script\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:06:25.588909 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-cni-netd\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.588909 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-modprobe-d\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.588909 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-host\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.588909 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588797    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-tuned\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.588909 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588822    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fdb73b3-0430-4d0c-a813-50a1beebdb12-tmp\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.588909 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-hostroot\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.588909 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-etc-kubernetes\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b-agent-certs\") pod \"konnectivity-agent-x4wzm\" (UID: \"17bd527b-9dd9-4fc6-86b3-57b5456c5a6b\") " pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-systemd-units\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.588987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkbch\" (UniqueName: \"kubernetes.io/projected/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-kube-api-access-nkbch\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-var-lib-kubelet\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-cni-multus\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-env-overrides\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589057    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-cni-binary-copy\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589078    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589115    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-sys-fs\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589184    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-sys\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589211    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-socket-dir-parent\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-multus-certs\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589270    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-slash\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-ovn\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.589492 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-node-log\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-log-socket\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589424    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589447    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-system-cni-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-kubelet\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d35c8150-ce21-4660-ad5d-e5cc1fce684b-dbus\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovn-node-metrics-cert\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-cnibin\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589553    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tktxh\" (UniqueName: \"kubernetes.io/projected/b2e85a91-0400-4117-a221-cc7961fd3ba0-kube-api-access-tktxh\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589577    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-cni-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-systemd\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589625    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-var-lib-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589664    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-cni-bin\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589705    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lclzk\" (UniqueName: \"kubernetes.io/projected/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-kube-api-access-lclzk\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589728    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysctl-d\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-serviceca\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:06:25.590258 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-socket-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-lib-modules\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589822    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-host-slash\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94qnj\" (UniqueName: \"kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj\") pod \"network-check-target-mr7hm\" (UID: \"ea79f848-e58a-463b-8c7f-d6f6dd0125b3\") " pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-hosts-file\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysconfig\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.589999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b-konnectivity-ca\") pod \"konnectivity-agent-x4wzm\" (UID: \"17bd527b-9dd9-4fc6-86b3-57b5456c5a6b\") " pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.590023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.590049    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovnkube-config\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.590073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovnkube-script-lib\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.590099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-host\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.590121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-run\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.590149    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-etc-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.590172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnds5\" (UniqueName: \"kubernetes.io/projected/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-kube-api-access-rnds5\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.590198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.590756 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.590226    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8c5k\" (UniqueName: \"kubernetes.io/projected/fad83c1a-3d12-44d3-9164-4c015d044f3e-kube-api-access-r8c5k\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.591241 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.590262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf4cc\" (UniqueName: \"kubernetes.io/projected/8fdb73b3-0430-4d0c-a813-50a1beebdb12-kube-api-access-sf4cc\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.599628 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.599599    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 03:06:25.599819 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.599801    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 03:06:25.599926 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.599859    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 03:06:25.599926 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.599902    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 03:06:25.600104 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.600074    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 03:06:25.600148 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.600102    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 03:06:25.604588 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.604568    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 03:06:25.604650 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.604574    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 03:06:25.608590 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.608569    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l8z7f\""
Oct 31 03:06:25.608676 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.608664    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bdmwz\""
Oct 31 03:06:25.608758 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.608693    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9g6gp\""
Oct 31 03:06:25.608758 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.608738    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 03:06:25.608907 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.608867    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 03:06:25.608907 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.608874    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 03:06:25.609050 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.608953    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2wckk\""
Oct 31 03:06:25.609050 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.608987    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:06:25.609050 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.609030    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 03:06:25.609050 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.609035    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 03:06:25.609206 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.609100    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-47gs9\""
Oct 31 03:06:25.609549 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.609536    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gtdck\""
Oct 31 03:06:25.611135 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.611118    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-95c6m\""
Oct 31 03:06:25.649973 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.649937    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 03:06:25.649973 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.649937    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 03:06:25.650302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.650086    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 03:06:25.650302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.650205    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 03:06:25.650302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.650275    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 03:06:25.650477 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.650369    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 03:06:25.650477 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.650443    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xgll4"
Oct 31 03:06:25.690603 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.690547    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-socket-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.690791 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.690603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-lib-modules\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.690791 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.690715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:25.690791 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.690746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-host-slash\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:06:25.690791 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.690766    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-lib-modules\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.690791 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.690773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qnj\" (UniqueName: \"kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj\") pod \"network-check-target-mr7hm\" (UID: \"ea79f848-e58a-463b-8c7f-d6f6dd0125b3\") " pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:25.691090 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.690814    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-socket-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.691090 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.690842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-hosts-file\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:06:25.691090 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.690866    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-host-slash\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:06:25.691090 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:25.690914    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:25.691090 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.690921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysconfig\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.691090 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.690952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b-konnectivity-ca\") pod \"konnectivity-agent-x4wzm\" (UID: \"17bd527b-9dd9-4fc6-86b3-57b5456c5a6b\") " pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:06:25.691090 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.690938    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-hosts-file\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:06:25.691090 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:25.691001    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret podName:d35c8150-ce21-4660-ad5d-e5cc1fce684b nodeName:}" failed. No retries permitted until 2026-10-31 03:06:26.190973906 +0000 UTC m=+3.238936096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret") pod "global-pull-secret-syncer-ldcbk" (UID: "d35c8150-ce21-4660-ad5d-e5cc1fce684b") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:25.691090 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691000    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysconfig\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.691090 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.691090 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691053    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.691090 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovnkube-config\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovnkube-script-lib\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-host\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-run\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-etc-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnds5\" (UniqueName: \"kubernetes.io/projected/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-kube-api-access-rnds5\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691385    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8c5k\" (UniqueName: \"kubernetes.io/projected/fad83c1a-3d12-44d3-9164-4c015d044f3e-kube-api-access-r8c5k\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691453    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-etc-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691482    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-run\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4cc\" (UniqueName: \"kubernetes.io/projected/8fdb73b3-0430-4d0c-a813-50a1beebdb12-kube-api-access-sf4cc\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysctl-conf\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d35c8150-ce21-4660-ad5d-e5cc1fce684b-kubelet-config\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-run-netns\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.691636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691643    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnpf\" (UniqueName: \"kubernetes.io/projected/0fbec67c-68ae-48f9-b780-bff7931da269-kube-api-access-4dnpf\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691670    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-registration-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691678    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysctl-conf\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-kubernetes\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691764    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-systemd\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691804    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691677    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b-konnectivity-ca\") pod \"konnectivity-agent-x4wzm\" (UID: \"17bd527b-9dd9-4fc6-86b3-57b5456c5a6b\") " pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-k8s-cni-cncf-io\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691847    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovnkube-config\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691854    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-kubernetes\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691871    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovnkube-script-lib\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691774    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-registration-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691886    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-systemd\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-conf-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691939    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-run-netns\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-daemon-config\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691981    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-kubelet\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.692302 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.691987    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-k8s-cni-cncf-io\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692006    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d35c8150-ce21-4660-ad5d-e5cc1fce684b-kubelet-config\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-cni-bin\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692131    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-kubelet\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692135    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-host\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692163    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-cni-bin\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692188    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-conf-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d78tf\" (UniqueName: \"kubernetes.io/projected/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-kube-api-access-d78tf\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:25.692337    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692403    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692493    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-daemon-config\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:25.692505    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs podName:0fbec67c-68ae-48f9-b780-bff7931da269 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:26.192481083 +0000 UTC m=+3.240443250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs") pod "network-metrics-daemon-sl5hn" (UID: "0fbec67c-68ae-48f9-b780-bff7931da269") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692547    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-system-cni-dir\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692587    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-system-cni-dir\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692591    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.692847 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-device-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-tmp-dir\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-cnibin\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692734    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-device-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/382f0f50-a8e4-463d-a64c-710cfffdc963-cni-binary-copy\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692711    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-netns\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-netns\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692781    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknn7\" (UniqueName: \"kubernetes.io/projected/382f0f50-a8e4-463d-a64c-710cfffdc963-kube-api-access-tknn7\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692832    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-cnibin\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-iptables-alerter-script\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-cni-netd\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-modprobe-d\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.692963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-host\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693004    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-cni-netd\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693087    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-tmp-dir\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:06:25.693658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693102    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-tuned\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693117    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-modprobe-d\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fdb73b3-0430-4d0c-a813-50a1beebdb12-tmp\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693165    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-host\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-hostroot\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693218    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-etc-kubernetes\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693225    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-hostroot\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693265    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b-agent-certs\") pod \"konnectivity-agent-x4wzm\" (UID: \"17bd527b-9dd9-4fc6-86b3-57b5456c5a6b\") " pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693313    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/382f0f50-a8e4-463d-a64c-710cfffdc963-cni-binary-copy\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-systemd-units\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693377    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkbch\" (UniqueName: \"kubernetes.io/projected/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-kube-api-access-nkbch\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693381    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-systemd-units\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693439    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-var-lib-kubelet\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693454    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-iptables-alerter-script\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693479    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-var-lib-kubelet\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693496    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-cni-multus\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693481    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-cni-multus\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693477    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 03:06:25.694518 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-env-overrides\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693595    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-etc-kubernetes\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-cni-binary-copy\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.693998    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-sys-fs\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-sys\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694062    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-env-overrides\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-socket-dir-parent\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694169    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-sys\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694232    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-sys-fs\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694292    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-socket-dir-parent\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694427    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-multus-certs\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694585    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-multus-certs\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-slash\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-ovn\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-node-log\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.695383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-log-socket\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694679    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-system-cni-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694712    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-kubelet\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d35c8150-ce21-4660-ad5d-e5cc1fce684b-dbus\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694737    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-log-socket\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694726    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-ovn\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovn-node-metrics-cert\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694799    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-slash\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-node-log\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694928    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694938    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-kubelet\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.694982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-cnibin\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695011    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-system-cni-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695026    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tktxh\" (UniqueName: \"kubernetes.io/projected/b2e85a91-0400-4117-a221-cc7961fd3ba0-kube-api-access-tktxh\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695058    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-cnibin\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.696273 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695113    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-cni-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695167    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d35c8150-ce21-4660-ad5d-e5cc1fce684b-dbus\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695306    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-systemd\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695345    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-cni-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695356    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695380    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-var-lib-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695396    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-systemd\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695400    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-cni-binary-copy\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-cni-bin\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695515    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-var-lib-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lclzk\" (UniqueName: \"kubernetes.io/projected/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-kube-api-access-lclzk\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695551    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-cni-bin\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695587    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysctl-d\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-serviceca\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.695758    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysctl-d\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.697143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.696100    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-serviceca\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:06:25.697813 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.697620    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fdb73b3-0430-4d0c-a813-50a1beebdb12-tmp\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.697813 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.697659    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-tuned\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.697813 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.697765    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b-agent-certs\") pod \"konnectivity-agent-x4wzm\" (UID: \"17bd527b-9dd9-4fc6-86b3-57b5456c5a6b\") " pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:06:25.697906 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.697858    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovn-node-metrics-cert\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.748070 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:25.748032    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:25.748070 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:25.748072    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:25.748420 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:25.748091    2576 projected.go:196] Error preparing data for projected volume kube-api-access-94qnj for pod openshift-network-diagnostics/network-check-target-mr7hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:25.748420 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:25.748174    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj podName:ea79f848-e58a-463b-8c7f-d6f6dd0125b3 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:26.248147526 +0000 UTC m=+3.296109712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-94qnj" (UniqueName: "kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj") pod "network-check-target-mr7hm" (UID: "ea79f848-e58a-463b-8c7f-d6f6dd0125b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:25.751500 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.751450    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tknn7\" (UniqueName: \"kubernetes.io/projected/382f0f50-a8e4-463d-a64c-710cfffdc963-kube-api-access-tknn7\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.751737 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.751711    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4cc\" (UniqueName: \"kubernetes.io/projected/8fdb73b3-0430-4d0c-a813-50a1beebdb12-kube-api-access-sf4cc\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.751804 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.751738    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkbch\" (UniqueName: \"kubernetes.io/projected/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-kube-api-access-nkbch\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:06:25.751979 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.751935    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnds5\" (UniqueName: \"kubernetes.io/projected/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-kube-api-access-rnds5\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:06:25.752072 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.751913    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnpf\" (UniqueName: \"kubernetes.io/projected/0fbec67c-68ae-48f9-b780-bff7931da269-kube-api-access-4dnpf\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:25.752293 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.752271    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d78tf\" (UniqueName: \"kubernetes.io/projected/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-kube-api-access-d78tf\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:06:25.752434 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.752411    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8c5k\" (UniqueName: \"kubernetes.io/projected/fad83c1a-3d12-44d3-9164-4c015d044f3e-kube-api-access-r8c5k\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.752495 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.752412    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tktxh\" (UniqueName: \"kubernetes.io/projected/b2e85a91-0400-4117-a221-cc7961fd3ba0-kube-api-access-tktxh\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.753399 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.753333    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lclzk\" (UniqueName: \"kubernetes.io/projected/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-kube-api-access-lclzk\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.813261 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.813220    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:25.817708 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.817679    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-68wpw"
Oct 31 03:06:25.824535 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.824505    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:06:25.828105 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:25.828069    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382f0f50_a8e4_463d_a64c_710cfffdc963.slice/crio-f2e709eb582e6a911923218b63ea491261856e717672fb11192fe81149fdd15c WatchSource:0}: Error finding container f2e709eb582e6a911923218b63ea491261856e717672fb11192fe81149fdd15c: Status 404 returned error can't find the container with id f2e709eb582e6a911923218b63ea491261856e717672fb11192fe81149fdd15c
Oct 31 03:06:25.832792 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.832763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:06:25.833092 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:25.833066    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd79b48_c38e_4822_86b5_4b7c3c858fb6.slice/crio-add7b8d0710d0f7e7e60b81bbc250a61606f4fc694358c7f67940db9debef870 WatchSource:0}: Error finding container add7b8d0710d0f7e7e60b81bbc250a61606f4fc694358c7f67940db9debef870: Status 404 returned error can't find the container with id add7b8d0710d0f7e7e60b81bbc250a61606f4fc694358c7f67940db9debef870
Oct 31 03:06:25.839691 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.839670    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:06:25.841305 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:25.841278    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20dcc2be_01c6_4294_bce6_3b9a77f0d3d9.slice/crio-622440e093a3efaedfc3d60aabc00f5694e0f8d0981adecfad62281f028e47c3 WatchSource:0}: Error finding container 622440e093a3efaedfc3d60aabc00f5694e0f8d0981adecfad62281f028e47c3: Status 404 returned error can't find the container with id 622440e093a3efaedfc3d60aabc00f5694e0f8d0981adecfad62281f028e47c3
Oct 31 03:06:25.848634 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:25.848604    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bd527b_9dd9_4fc6_86b3_57b5456c5a6b.slice/crio-8c52cad3c746ec9eff866d2b783fca4e2103ec9d11e7b1006c24f1873aec35d5 WatchSource:0}: Error finding container 8c52cad3c746ec9eff866d2b783fca4e2103ec9d11e7b1006c24f1873aec35d5: Status 404 returned error can't find the container with id 8c52cad3c746ec9eff866d2b783fca4e2103ec9d11e7b1006c24f1873aec35d5
Oct 31 03:06:25.854126 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.854104    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:06:25.862394 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:25.862366    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e85a91_0400_4117_a221_cc7961fd3ba0.slice/crio-1559f59a3675c2e3d2c6133bb4ebf1f36a6d8183bb017536f51686ab1fb3e1b5 WatchSource:0}: Error finding container 1559f59a3675c2e3d2c6133bb4ebf1f36a6d8183bb017536f51686ab1fb3e1b5: Status 404 returned error can't find the container with id 1559f59a3675c2e3d2c6133bb4ebf1f36a6d8183bb017536f51686ab1fb3e1b5
Oct 31 03:06:25.868008 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.867984    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:06:25.869044 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.869026    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:06:25.871784 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.871759    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:25.873101 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:25.873078    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:06:25.884722 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:25.884675    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af7ee9f_95f0_4632_989a_2beaf0fba3e3.slice/crio-ca9219ff27b1e5924dbd2574c0c2526c859a7b2880c484e6092d27630adb2dd0 WatchSource:0}: Error finding container ca9219ff27b1e5924dbd2574c0c2526c859a7b2880c484e6092d27630adb2dd0: Status 404 returned error can't find the container with id ca9219ff27b1e5924dbd2574c0c2526c859a7b2880c484e6092d27630adb2dd0
Oct 31 03:06:25.887389 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:06:25.887365    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fdb73b3_0430_4d0c_a813_50a1beebdb12.slice/crio-71c8ea186ff2b28e080fe69baee70f1bfc94eeb1d0d0a6a5a9d2ba1895728649 WatchSource:0}: Error finding container 71c8ea186ff2b28e080fe69baee70f1bfc94eeb1d0d0a6a5a9d2ba1895728649: Status 404 returned error can't find the container with id 71c8ea186ff2b28e080fe69baee70f1bfc94eeb1d0d0a6a5a9d2ba1895728649
Oct 31 03:06:26.199187 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.199088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:26.199187 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.199163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:26.199443 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:26.199203    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:26.199443 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:26.199278    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret podName:d35c8150-ce21-4660-ad5d-e5cc1fce684b nodeName:}" failed. No retries permitted until 2026-10-31 03:06:27.199258699 +0000 UTC m=+4.247220866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret") pod "global-pull-secret-syncer-ldcbk" (UID: "d35c8150-ce21-4660-ad5d-e5cc1fce684b") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:26.199443 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:26.199298    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:26.199443 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:26.199350    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs podName:0fbec67c-68ae-48f9-b780-bff7931da269 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:27.199334008 +0000 UTC m=+4.247296175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs") pod "network-metrics-daemon-sl5hn" (UID: "0fbec67c-68ae-48f9-b780-bff7931da269") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:26.300409 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.300366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qnj\" (UniqueName: \"kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj\") pod \"network-check-target-mr7hm\" (UID: \"ea79f848-e58a-463b-8c7f-d6f6dd0125b3\") " pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:26.300621 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:26.300521    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:26.300621 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:26.300542    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:26.300621 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:26.300556    2576 projected.go:196] Error preparing data for projected volume kube-api-access-94qnj for pod openshift-network-diagnostics/network-check-target-mr7hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:26.300621 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:26.300617    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj podName:ea79f848-e58a-463b-8c7f-d6f6dd0125b3 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:27.30060129 +0000 UTC m=+4.348563458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-94qnj" (UniqueName: "kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj") pod "network-check-target-mr7hm" (UID: "ea79f848-e58a-463b-8c7f-d6f6dd0125b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:26.524932 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.524807    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-68wpw" event={"ID":"382f0f50-a8e4-463d-a64c-710cfffdc963","Type":"ContainerStarted","Data":"f2e709eb582e6a911923218b63ea491261856e717672fb11192fe81149fdd15c"}
Oct 31 03:06:26.526434 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.526392    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws" event={"ID":"0af7ee9f-95f0-4632-989a-2beaf0fba3e3","Type":"ContainerStarted","Data":"ca9219ff27b1e5924dbd2574c0c2526c859a7b2880c484e6092d27630adb2dd0"}
Oct 31 03:06:26.527827 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.527791    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fbw9q" event={"ID":"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a","Type":"ContainerStarted","Data":"94a2cdb3b61e63680197f9105bfda11dc1cc8e2dd657361fc4382ddaab4fcc67"}
Oct 31 03:06:26.529390 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.529333    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x4wzm" event={"ID":"17bd527b-9dd9-4fc6-86b3-57b5456c5a6b","Type":"ContainerStarted","Data":"8c52cad3c746ec9eff866d2b783fca4e2103ec9d11e7b1006c24f1873aec35d5"}
Oct 31 03:06:26.530889 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.530860    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dkpt8" event={"ID":"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9","Type":"ContainerStarted","Data":"622440e093a3efaedfc3d60aabc00f5694e0f8d0981adecfad62281f028e47c3"}
Oct 31 03:06:26.532572 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.532350    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bzwqk" event={"ID":"4cd79b48-c38e-4822-86b5-4b7c3c858fb6","Type":"ContainerStarted","Data":"add7b8d0710d0f7e7e60b81bbc250a61606f4fc694358c7f67940db9debef870"}
Oct 31 03:06:26.533661 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.533626    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9775g" event={"ID":"8fdb73b3-0430-4d0c-a813-50a1beebdb12","Type":"ContainerStarted","Data":"71c8ea186ff2b28e080fe69baee70f1bfc94eeb1d0d0a6a5a9d2ba1895728649"}
Oct 31 03:06:26.534936 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.534909    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj" event={"ID":"fad83c1a-3d12-44d3-9164-4c015d044f3e","Type":"ContainerStarted","Data":"b73281d3fe33cea4e1073bb7e61c6266744c827e96a7af1e468af3cbb416f2f1"}
Oct 31 03:06:26.536286 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.536260    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh7cw" event={"ID":"b2e85a91-0400-4117-a221-cc7961fd3ba0","Type":"ContainerStarted","Data":"1559f59a3675c2e3d2c6133bb4ebf1f36a6d8183bb017536f51686ab1fb3e1b5"}
Oct 31 03:06:26.653058 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.653004    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:01:25 +0000 UTC" deadline="2028-07-04 03:58:32.550296298 +0000 UTC"
Oct 31 03:06:26.653058 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:26.653051    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14688h52m5.897250013s"
Oct 31 03:06:27.210052 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:27.210005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:27.210220 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:27.210107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:27.210332 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:27.210286    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:27.210371 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:27.210360    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs podName:0fbec67c-68ae-48f9-b780-bff7931da269 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:29.210338683 +0000 UTC m=+6.258300853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs") pod "network-metrics-daemon-sl5hn" (UID: "0fbec67c-68ae-48f9-b780-bff7931da269") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:27.210488 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:27.210472    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:27.210543 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:27.210521    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret podName:d35c8150-ce21-4660-ad5d-e5cc1fce684b nodeName:}" failed. No retries permitted until 2026-10-31 03:06:29.210508104 +0000 UTC m=+6.258470290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret") pod "global-pull-secret-syncer-ldcbk" (UID: "d35c8150-ce21-4660-ad5d-e5cc1fce684b") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:27.311441 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:27.311360    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qnj\" (UniqueName: \"kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj\") pod \"network-check-target-mr7hm\" (UID: \"ea79f848-e58a-463b-8c7f-d6f6dd0125b3\") " pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:27.311648 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:27.311596    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:27.311648 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:27.311623    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:27.311648 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:27.311637    2576 projected.go:196] Error preparing data for projected volume kube-api-access-94qnj for pod openshift-network-diagnostics/network-check-target-mr7hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:27.311794 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:27.311702    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj podName:ea79f848-e58a-463b-8c7f-d6f6dd0125b3 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:29.311683876 +0000 UTC m=+6.359646060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-94qnj" (UniqueName: "kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj") pod "network-check-target-mr7hm" (UID: "ea79f848-e58a-463b-8c7f-d6f6dd0125b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:27.515314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:27.515275    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:27.515504 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:27.515426    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:27.515582 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:27.515524    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:27.515648 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:27.515611    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:27.515711 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:27.515687    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:27.515802 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:27.515779    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:27.653838 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:27.653761    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:01:25 +0000 UTC" deadline="2028-04-12 21:12:04.975646764 +0000 UTC"
Oct 31 03:06:27.653838 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:27.653833    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12714h5m37.321817876s"
Oct 31 03:06:29.230125 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:29.229817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:29.230125 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:29.230119    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:29.230831 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:29.230187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:29.230831 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:29.230215    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs podName:0fbec67c-68ae-48f9-b780-bff7931da269 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:33.23018876 +0000 UTC m=+10.278150949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs") pod "network-metrics-daemon-sl5hn" (UID: "0fbec67c-68ae-48f9-b780-bff7931da269") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:29.230831 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:29.230339    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:29.230831 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:29.230398    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret podName:d35c8150-ce21-4660-ad5d-e5cc1fce684b nodeName:}" failed. No retries permitted until 2026-10-31 03:06:33.230380801 +0000 UTC m=+10.278342984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret") pod "global-pull-secret-syncer-ldcbk" (UID: "d35c8150-ce21-4660-ad5d-e5cc1fce684b") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:29.330662 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:29.330627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qnj\" (UniqueName: \"kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj\") pod \"network-check-target-mr7hm\" (UID: \"ea79f848-e58a-463b-8c7f-d6f6dd0125b3\") " pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:29.330863 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:29.330839    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:29.330863 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:29.330863    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:29.331022 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:29.330877    2576 projected.go:196] Error preparing data for projected volume kube-api-access-94qnj for pod openshift-network-diagnostics/network-check-target-mr7hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:29.331022 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:29.330969    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj podName:ea79f848-e58a-463b-8c7f-d6f6dd0125b3 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:33.330947078 +0000 UTC m=+10.378909263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-94qnj" (UniqueName: "kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj") pod "network-check-target-mr7hm" (UID: "ea79f848-e58a-463b-8c7f-d6f6dd0125b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:29.515012 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:29.514524    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:29.515012 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:29.514678    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:29.515012 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:29.514749    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:29.515012 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:29.514824    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:29.515012 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:29.514874    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:29.515012 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:29.514972    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:29.554520 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:29.553728    2576 generic.go:356] "Generic (PLEG): container finished" podID="41bbbd7b4d4d448be0c950dd63a0f8ce" containerID="d902b90358cd0471f6e43bbd45900acbb62c343711c3d7d585ec9083e368d245" exitCode=0
Oct 31 03:06:29.554520 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:29.553814    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal" event={"ID":"41bbbd7b4d4d448be0c950dd63a0f8ce","Type":"ContainerDied","Data":"d902b90358cd0471f6e43bbd45900acbb62c343711c3d7d585ec9083e368d245"}
Oct 31 03:06:29.564377 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:29.563725    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh7cw" event={"ID":"b2e85a91-0400-4117-a221-cc7961fd3ba0","Type":"ContainerStarted","Data":"8adf7fe9742b3c591bc35a2ca447f55415a078c6f0e495b6addda94645cabaeb"}
Oct 31 03:06:30.579387 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:30.579338    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal" event={"ID":"41bbbd7b4d4d448be0c950dd63a0f8ce","Type":"ContainerStarted","Data":"78de8ed21133c98abb9e253f8616398f413c92748eebc5e316beeb74687f9927"}
Oct 31 03:06:30.584456 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:30.583525    2576 generic.go:356] "Generic (PLEG): container finished" podID="b2e85a91-0400-4117-a221-cc7961fd3ba0" containerID="8adf7fe9742b3c591bc35a2ca447f55415a078c6f0e495b6addda94645cabaeb" exitCode=0
Oct 31 03:06:30.584456 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:30.583589    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh7cw" event={"ID":"b2e85a91-0400-4117-a221-cc7961fd3ba0","Type":"ContainerDied","Data":"8adf7fe9742b3c591bc35a2ca447f55415a078c6f0e495b6addda94645cabaeb"}
Oct 31 03:06:30.697133 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:30.696386    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal" podStartSLOduration=5.69636523 podStartE2EDuration="5.69636523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:30.695427715 +0000 UTC m=+7.743389907" watchObservedRunningTime="2026-10-31 03:06:30.69636523 +0000 UTC m=+7.744327423"
Oct 31 03:06:31.515231 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:31.514996    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:31.515231 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:31.515050    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:31.515231 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:31.515149    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:31.515544 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:31.515296    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:31.515544 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:31.515333    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:31.515544 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:31.515401    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:31.589174 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:31.589132    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x4wzm" event={"ID":"17bd527b-9dd9-4fc6-86b3-57b5456c5a6b","Type":"ContainerStarted","Data":"cb6661c297250723c86df0237103b9b247f2a38e30f7d75a380570a704cb5f99"}
Oct 31 03:06:31.593747 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:31.593716    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal" event={"ID":"d510a50aad744fb4754386c4136c4a71","Type":"ContainerStarted","Data":"d3b4e3d5797ab6c6cf1652b19ff83f3885cd9cba23e721ca31528cd0b2ed7a08"}
Oct 31 03:06:31.803501 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:31.803040    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x4wzm" podStartSLOduration=2.9524107 podStartE2EDuration="7.80301852s" totalImagesPullingTime="4.85060782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:31.697098887 +0000 UTC m=+8.745061078" watchObservedRunningTime="2026-10-31 03:06:31.80301852 +0000 UTC m=+8.850980911"
Oct 31 03:06:32.598210 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:32.598123    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj" event={"ID":"fad83c1a-3d12-44d3-9164-4c015d044f3e","Type":"ContainerStarted","Data":"7ce07ffb0006f00a7a880e2259565c8f0fc5255decb372bb00c5e816f7bfed34"}
Oct 31 03:06:33.269408 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:33.268937    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:33.269408 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:33.269024    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:33.269408 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:33.269158    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:33.269408 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:33.269222    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs podName:0fbec67c-68ae-48f9-b780-bff7931da269 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.269204503 +0000 UTC m=+18.317166675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs") pod "network-metrics-daemon-sl5hn" (UID: "0fbec67c-68ae-48f9-b780-bff7931da269") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:33.269408 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:33.269284    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:33.269408 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:33.269316    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret podName:d35c8150-ce21-4660-ad5d-e5cc1fce684b nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.269305936 +0000 UTC m=+18.317268115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret") pod "global-pull-secret-syncer-ldcbk" (UID: "d35c8150-ce21-4660-ad5d-e5cc1fce684b") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:33.370495 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:33.370454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qnj\" (UniqueName: \"kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj\") pod \"network-check-target-mr7hm\" (UID: \"ea79f848-e58a-463b-8c7f-d6f6dd0125b3\") " pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:33.370712 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:33.370682    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:33.370712 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:33.370712    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:33.370909 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:33.370726    2576 projected.go:196] Error preparing data for projected volume kube-api-access-94qnj for pod openshift-network-diagnostics/network-check-target-mr7hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:33.370909 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:33.370792    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj podName:ea79f848-e58a-463b-8c7f-d6f6dd0125b3 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.37077044 +0000 UTC m=+18.418732620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-94qnj" (UniqueName: "kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj") pod "network-check-target-mr7hm" (UID: "ea79f848-e58a-463b-8c7f-d6f6dd0125b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:33.516514 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:33.516035    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:33.516514 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:33.516169    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:33.516514 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:33.516247    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:33.516514 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:33.516312    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:33.516514 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:33.516375    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:33.516514 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:33.516457    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:33.603423 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:33.603323    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bzwqk" event={"ID":"4cd79b48-c38e-4822-86b5-4b7c3c858fb6","Type":"ContainerStarted","Data":"9955aa0b7ec8fbd06c4c4a2d38cd0146c7d932eb4aa0f937c12fc8da0b0f34b2"}
Oct 31 03:06:33.709321 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:33.708097    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal" podStartSLOduration=8.708078734 podStartE2EDuration="8.708078734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:31.803005103 +0000 UTC m=+8.850967292" watchObservedRunningTime="2026-10-31 03:06:33.708078734 +0000 UTC m=+10.756040913"
Oct 31 03:06:33.744432 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:33.744395    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:06:34.647974 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:34.647936    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:06:34.649417 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:34.649394    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:06:34.755185 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:34.755126    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bzwqk" podStartSLOduration=3.860958151 podStartE2EDuration="10.755105411s" totalImagesPullingTime="6.89414726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:33.709300294 +0000 UTC m=+10.757262488" watchObservedRunningTime="2026-10-31 03:06:34.755105411 +0000 UTC m=+11.803067601"
Oct 31 03:06:35.515205 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:35.514991    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:35.515405 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:35.514994    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:35.515405 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:35.515320    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:35.515405 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:35.515022    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:35.515519 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:35.515399    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:35.515563 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:35.515511    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:35.611734 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:35.611698    2576 generic.go:356] "Generic (PLEG): container finished" podID="b2e85a91-0400-4117-a221-cc7961fd3ba0" containerID="ca65a0863a030e35121881af8cc9cdd6f28c6c4881a4b10fa85e175ebb28ca6a" exitCode=0
Oct 31 03:06:35.611930 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:35.611786    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh7cw" event={"ID":"b2e85a91-0400-4117-a221-cc7961fd3ba0","Type":"ContainerDied","Data":"ca65a0863a030e35121881af8cc9cdd6f28c6c4881a4b10fa85e175ebb28ca6a"}
Oct 31 03:06:35.616054 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:35.616021    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-68wpw" event={"ID":"382f0f50-a8e4-463d-a64c-710cfffdc963","Type":"ContainerStarted","Data":"bd21027084cd58f330d9877bd33cee0129a1f159c190ca552db06ae5efe587b3"}
Oct 31 03:06:35.616458 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:35.616439    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:06:35.934304 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:35.934190    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-68wpw" podStartSLOduration=2.981926621 podStartE2EDuration="11.934167164s" totalImagesPullingTime="8.952240543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:35.933064434 +0000 UTC m=+12.981026634" watchObservedRunningTime="2026-10-31 03:06:35.934167164 +0000 UTC m=+12.982129355"
Oct 31 03:06:36.618938 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:36.618825    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9775g" event={"ID":"8fdb73b3-0430-4d0c-a813-50a1beebdb12","Type":"ContainerStarted","Data":"f0ce2a0aa9aa06ea36832c04247a74ab80cab9e4e099166b45aa50a916e9415d"}
Oct 31 03:06:36.620520 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:36.620484    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fbw9q" event={"ID":"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a","Type":"ContainerStarted","Data":"4a0c60ec8a1a5398efdace9a4618fe6b13e613e928732cabb5c4d4190cddc248"}
Oct 31 03:06:36.727026 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:36.726965    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9775g" podStartSLOduration=2.470655165 podStartE2EDuration="12.726944319s" totalImagesPullingTime="10.256289154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:36.726079386 +0000 UTC m=+13.774041574" watchObservedRunningTime="2026-10-31 03:06:36.726944319 +0000 UTC m=+13.774906509"
Oct 31 03:06:36.831720 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:36.831659    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-fbw9q" podStartSLOduration=2.585072076 podStartE2EDuration="12.831645017s" totalImagesPullingTime="10.246572941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:36.831273918 +0000 UTC m=+13.879236109" watchObservedRunningTime="2026-10-31 03:06:36.831645017 +0000 UTC m=+13.879607204"
Oct 31 03:06:37.512299 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:37.512258    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:37.512299 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:37.512291    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:37.513287 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:37.512436    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:37.513287 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:37.512476    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:37.513287 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:37.512556    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:37.513287 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:37.512613    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:37.624868 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:37.624833    2576 generic.go:356] "Generic (PLEG): container finished" podID="b2e85a91-0400-4117-a221-cc7961fd3ba0" containerID="73dde071cd64b0d743fc742b089fcd95ed26b87008c5763123df758fd23a86d1" exitCode=0
Oct 31 03:06:37.625039 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:37.624915    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh7cw" event={"ID":"b2e85a91-0400-4117-a221-cc7961fd3ba0","Type":"ContainerDied","Data":"73dde071cd64b0d743fc742b089fcd95ed26b87008c5763123df758fd23a86d1"}
Oct 31 03:06:39.513018 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:39.512984    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:39.513532 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:39.513003    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:39.513532 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:39.513106    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:39.513532 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:39.513125    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:39.513532 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:39.513206    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:39.513532 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:39.513306    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:39.631171 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:39.631123    2576 generic.go:356] "Generic (PLEG): container finished" podID="b2e85a91-0400-4117-a221-cc7961fd3ba0" containerID="2910743bf264e16e55f702a7ffcf56b3c1d737e8b89249f387feace41dd2747f" exitCode=0
Oct 31 03:06:39.631343 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:39.631186    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh7cw" event={"ID":"b2e85a91-0400-4117-a221-cc7961fd3ba0","Type":"ContainerDied","Data":"2910743bf264e16e55f702a7ffcf56b3c1d737e8b89249f387feace41dd2747f"}
Oct 31 03:06:41.328160 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:41.327944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:41.328627 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:41.328208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:41.328627 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:41.328118    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:41.328627 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:41.328297    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs podName:0fbec67c-68ae-48f9-b780-bff7931da269 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:57.328275473 +0000 UTC m=+34.376237653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs") pod "network-metrics-daemon-sl5hn" (UID: "0fbec67c-68ae-48f9-b780-bff7931da269") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:41.328627 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:41.328340    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:41.328627 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:41.328390    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret podName:d35c8150-ce21-4660-ad5d-e5cc1fce684b nodeName:}" failed. No retries permitted until 2026-10-31 03:06:57.328379443 +0000 UTC m=+34.376341609 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret") pod "global-pull-secret-syncer-ldcbk" (UID: "d35c8150-ce21-4660-ad5d-e5cc1fce684b") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:41.428730 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:41.428686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qnj\" (UniqueName: \"kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj\") pod \"network-check-target-mr7hm\" (UID: \"ea79f848-e58a-463b-8c7f-d6f6dd0125b3\") " pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:41.428945 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:41.428861    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:41.428945 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:41.428897    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:41.428945 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:41.428910    2576 projected.go:196] Error preparing data for projected volume kube-api-access-94qnj for pod openshift-network-diagnostics/network-check-target-mr7hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:41.429086 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:41.428973    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj podName:ea79f848-e58a-463b-8c7f-d6f6dd0125b3 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:57.428954951 +0000 UTC m=+34.476917130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-94qnj" (UniqueName: "kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj") pod "network-check-target-mr7hm" (UID: "ea79f848-e58a-463b-8c7f-d6f6dd0125b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:41.512743 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:41.512660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:41.512743 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:41.512709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:41.512997 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:41.512670    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:41.512997 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:41.512797    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:41.512997 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:41.512912    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:41.513138 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:41.513021    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:42.639833 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:42.639759    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dkpt8" event={"ID":"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9","Type":"ContainerStarted","Data":"6f9d8c87213999ed3cc911cf5a9be0df7f5f916345e443aadd95fda8cbbbf70d"}
Oct 31 03:06:42.746201 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:42.746146    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dkpt8" podStartSLOduration=4.047437176 podStartE2EDuration="18.746132091s" totalImagesPullingTime="14.698694915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:42.745858862 +0000 UTC m=+19.793821051" watchObservedRunningTime="2026-10-31 03:06:42.746132091 +0000 UTC m=+19.794094279"
Oct 31 03:06:43.056894 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.056819    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 03:06:43.466389 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.466280    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:06:43.056837323Z","UUID":"5b6ed5b2-e5a8-429c-9245-780a73d84d42","Handler":null,"Name":"","Endpoint":""}
Oct 31 03:06:43.467997 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.467973    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 03:06:43.467997 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.468003    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 03:06:43.513984 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.513954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:43.514132 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.514112    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:43.514205 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:43.514175    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:43.514316 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:43.514293    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:43.514379 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.514324    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:43.514442 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:43.514420    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:43.647448 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.647201    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws" event={"ID":"0af7ee9f-95f0-4632-989a-2beaf0fba3e3","Type":"ContainerStarted","Data":"39c3fd1ba15a61ad23c7a1ae2e24a000ac5423126eea3bb80f38024056f10def"}
Oct 31 03:06:43.648012 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.647512    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws" event={"ID":"0af7ee9f-95f0-4632-989a-2beaf0fba3e3","Type":"ContainerStarted","Data":"6b38d8aab9b3219807ad6db8e51f1f7e72fa57b8ee3ca951f06aff7148314fd1"}
Oct 31 03:06:43.648012 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.647563    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws" event={"ID":"0af7ee9f-95f0-4632-989a-2beaf0fba3e3","Type":"ContainerStarted","Data":"908a88933bf9c7d2e2fffe7c68a76b40eac3c45512f0daa7e0a6d46ebc71c2ad"}
Oct 31 03:06:43.648012 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.647615    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws" event={"ID":"0af7ee9f-95f0-4632-989a-2beaf0fba3e3","Type":"ContainerStarted","Data":"740dfbb9150776007bc7cc982dbcb61697ea42e1ba18581e5317cb20c5d36074"}
Oct 31 03:06:43.648012 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.647659    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws" event={"ID":"0af7ee9f-95f0-4632-989a-2beaf0fba3e3","Type":"ContainerStarted","Data":"dcc66f40e61236f0c0447e25ff122b5a83000c48ad249bd75bcfbad93d8e8763"}
Oct 31 03:06:43.648012 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.647701    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws" event={"ID":"0af7ee9f-95f0-4632-989a-2beaf0fba3e3","Type":"ContainerStarted","Data":"d5eb765ed74ca009e52200379bc2bb0e28fef526261fc287a30c34914e7f58c5"}
Oct 31 03:06:43.650906 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:43.650859    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj" event={"ID":"fad83c1a-3d12-44d3-9164-4c015d044f3e","Type":"ContainerStarted","Data":"bf13982814c78facd0268af90083548a7631148385b245f823b2ac95e2e04e11"}
Oct 31 03:06:44.656182 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:44.656140    2576 generic.go:356] "Generic (PLEG): container finished" podID="b2e85a91-0400-4117-a221-cc7961fd3ba0" containerID="182b1b72d287d8af70dd2f53d6758ab84f85671487ef9b8c995fa91259983248" exitCode=0
Oct 31 03:06:44.656791 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:44.656202    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh7cw" event={"ID":"b2e85a91-0400-4117-a221-cc7961fd3ba0","Type":"ContainerDied","Data":"182b1b72d287d8af70dd2f53d6758ab84f85671487ef9b8c995fa91259983248"}
Oct 31 03:06:45.512711 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:45.512667    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:45.512711 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:45.512686    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:45.512963 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:45.512793    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:45.512963 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:45.512833    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:45.512963 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:45.512897    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:45.513079 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:45.512998    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:45.659888 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:45.659850    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj" event={"ID":"fad83c1a-3d12-44d3-9164-4c015d044f3e","Type":"ContainerStarted","Data":"f7a459787314d18275dbbdf6929ee62bf6644addd332a873e1330c50bf8fd95d"}
Oct 31 03:06:45.663141 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:45.663114    2576 generic.go:356] "Generic (PLEG): container finished" podID="b2e85a91-0400-4117-a221-cc7961fd3ba0" containerID="e0c4df9401b0c678c4ee95c11ccc555c0ec12aaed3560c69982d9846c544a268" exitCode=0
Oct 31 03:06:45.663295 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:45.663160    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh7cw" event={"ID":"b2e85a91-0400-4117-a221-cc7961fd3ba0","Type":"ContainerDied","Data":"e0c4df9401b0c678c4ee95c11ccc555c0ec12aaed3560c69982d9846c544a268"}
Oct 31 03:06:45.881413 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:45.881341    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj" podStartSLOduration=3.321474407 podStartE2EDuration="21.881324813s" totalImagesPullingTime="18.559850406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:45.770827576 +0000 UTC m=+22.818789766" watchObservedRunningTime="2026-10-31 03:06:45.881324813 +0000 UTC m=+22.929287001"
Oct 31 03:06:46.668023 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:46.667984    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws" event={"ID":"0af7ee9f-95f0-4632-989a-2beaf0fba3e3","Type":"ContainerStarted","Data":"a7dd01f0e96bb80a88da9bc2967a5cac6d82e4dbf9130fa29629888dcef6698e"}
Oct 31 03:06:46.671193 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:46.671161    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh7cw" event={"ID":"b2e85a91-0400-4117-a221-cc7961fd3ba0","Type":"ContainerStarted","Data":"f0172b42db359b89246bc4b326bdc9d557b8609d2b85f8dcda1e3d5c9aeffb65"}
Oct 31 03:06:46.874445 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:46.874388    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dh7cw" podStartSLOduration=7.928254192 podStartE2EDuration="22.874371984s" totalImagesPullingTime="14.946117792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:46.828085981 +0000 UTC m=+23.876048169" watchObservedRunningTime="2026-10-31 03:06:46.874371984 +0000 UTC m=+23.922334171"
Oct 31 03:06:47.512324 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:47.512288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:47.512478 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:47.512342    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:47.512478 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:47.512359    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:47.512478 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:47.512447    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:47.512619 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:47.512505    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:47.512619 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:47.512564    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:48.680952 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:48.680658    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws" event={"ID":"0af7ee9f-95f0-4632-989a-2beaf0fba3e3","Type":"ContainerStarted","Data":"2d6d5a272b65c5f95dff4ff209941971e8c82558a38e1a9d09c4d03664a513bf"}
Oct 31 03:06:48.681489 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:48.681010    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:48.681489 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:48.681079    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:48.698288 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:48.698252    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:48.933811 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:48.933712    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws" podStartSLOduration=7.577430796 podStartE2EDuration="24.933698965s" totalImagesPullingTime="17.356268169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:48.887435528 +0000 UTC m=+25.935397716" watchObservedRunningTime="2026-10-31 03:06:48.933698965 +0000 UTC m=+25.981661153"
Oct 31 03:06:49.512378 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:49.512348    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:49.512557 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:49.512400    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:49.512557 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:49.512422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:49.512557 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:49.512522    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:49.512692 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:49.512603    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:49.512692 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:49.512654    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:49.683158 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:49.683127    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:49.698678 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:49.698652    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:06:50.448767 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:50.448676    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mr7hm"]
Oct 31 03:06:50.448942 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:50.448793    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:50.448942 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:50.448906    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:50.464945 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:50.464907    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sl5hn"]
Oct 31 03:06:50.465134 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:50.465051    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:50.465198 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:50.465177    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:50.465636 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:50.465605    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ldcbk"]
Oct 31 03:06:50.465712 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:50.465700    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:50.465793 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:50.465779    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:52.512526 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:52.512317    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:52.512984 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:52.512328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:52.512984 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:52.512615    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:52.512984 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:52.512341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:52.512984 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:52.512714    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:52.512984 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:52.512769    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:52.761529 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:52.761457    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fbw9q_e9c7f8e8-6c5e-4f69-bede-8e89d342d02a/dns-node-resolver/0.log"
Oct 31 03:06:53.599981 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:53.599954    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bzwqk_4cd79b48-c38e-4822-86b5-4b7c3c858fb6/node-ca/0.log"
Oct 31 03:06:54.512846 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:54.512798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:54.512846 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:54.512824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:54.513103 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:54.512941    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:54.513103 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:54.512992    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:54.513184 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:54.513093    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:54.513215 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:54.513181    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:56.512620 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:56.512578    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:56.512620 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:56.512600    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:56.513184 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:56.512700    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:56.513184 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:56.512750    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:56.513184 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:56.512826    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:56.513184 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:56.512918    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:06:57.347148 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:57.347115    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:57.347331 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:57.347169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:57.347331 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:57.347253    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:57.347331 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:57.347258    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:57.347331 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:57.347308    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs podName:0fbec67c-68ae-48f9-b780-bff7931da269 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:29.347294241 +0000 UTC m=+66.395256420 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs") pod "network-metrics-daemon-sl5hn" (UID: "0fbec67c-68ae-48f9-b780-bff7931da269") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:57.347331 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:57.347321    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret podName:d35c8150-ce21-4660-ad5d-e5cc1fce684b nodeName:}" failed. No retries permitted until 2026-10-31 03:07:29.347315151 +0000 UTC m=+66.395277317 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret") pod "global-pull-secret-syncer-ldcbk" (UID: "d35c8150-ce21-4660-ad5d-e5cc1fce684b") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:57.447912 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:57.447871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qnj\" (UniqueName: \"kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj\") pod \"network-check-target-mr7hm\" (UID: \"ea79f848-e58a-463b-8c7f-d6f6dd0125b3\") " pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:57.448180 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:57.448024    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:57.448180 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:57.448050    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:57.448180 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:57.448062    2576 projected.go:196] Error preparing data for projected volume kube-api-access-94qnj for pod openshift-network-diagnostics/network-check-target-mr7hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:57.448180 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:57.448109    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj podName:ea79f848-e58a-463b-8c7f-d6f6dd0125b3 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:29.448097148 +0000 UTC m=+66.496059314 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-94qnj" (UniqueName: "kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj") pod "network-check-target-mr7hm" (UID: "ea79f848-e58a-463b-8c7f-d6f6dd0125b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:58.512377 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:58.512335    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:06:58.512949 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:58.512404    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:06:58.512949 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:06:58.512436    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:06:58.512949 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:58.512519    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:06:58.512949 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:58.512595    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:06:58.512949 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:06:58.512632    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:07:00.512804 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:00.512768    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:07:00.513425 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:00.512768    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:00.513425 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:00.512768    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:07:00.513425 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:00.512902    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:07:00.513425 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:00.512954    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:07:00.513425 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:00.513035    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:07:02.512896 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:02.512838    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:07:02.513356 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:02.512922    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:07:02.513356 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:02.513018    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:02.513356 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:02.513031    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:07:02.513356 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:02.513109    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:07:02.513356 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:02.513174    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:07:04.512422 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:04.512381    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:07:04.512860 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:04.512480    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:07:04.512860 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:04.512490    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:07:04.512860 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:04.512511    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:04.512860 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:04.512616    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:07:04.512860 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:04.512670    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:07:06.513164 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:06.513124    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:07:06.513580 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:06.513187    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:06.513580 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:06.513193    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:07:06.513580 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:06.513284    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:07:06.513580 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:06.513399    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:07:06.513580 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:06.513457    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:07:08.512618 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.512580    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:07:08.512618 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.512618    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:08.513221 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.512624    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:07:08.513221 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:08.512710    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sl5hn" podUID="0fbec67c-68ae-48f9-b780-bff7931da269"
Oct 31 03:07:08.513221 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:08.512790    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldcbk" podUID="d35c8150-ce21-4660-ad5d-e5cc1fce684b"
Oct 31 03:07:08.513221 ip-10-0-139-164 kubenswrapper[2576]: E1031 03:07:08.512896    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7hm" podUID="ea79f848-e58a-463b-8c7f-d6f6dd0125b3"
Oct 31 03:07:08.729544 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.729467    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeReady"
Oct 31 03:07:08.729719 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.729584    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 03:07:08.838323 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.838279    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4kbs4"]
Oct 31 03:07:08.842930 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.842903    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8452g"]
Oct 31 03:07:08.843102 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.843032    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4kbs4"
Oct 31 03:07:08.845023 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.844998    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:08.850866 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.850833    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2bqmj"]
Oct 31 03:07:08.854495 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.854467    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:08.892441 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.892405    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 03:07:08.893421 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.893397    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 03:07:08.893533 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.893473    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m6gdv\""
Oct 31 03:07:08.897972 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.897935    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8452g"]
Oct 31 03:07:08.903088 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.903069    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 03:07:08.903367 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.903341    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 03:07:08.903454 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.903367    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 03:07:08.903454 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.903367    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 03:07:08.903837 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.903816    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4kbs4"]
Oct 31 03:07:08.924690 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.924646    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-crio-socket\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:08.924690 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.924692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e387f3b0-3991-4570-895b-8b59c01c8f36-cert\") pod \"ingress-canary-4kbs4\" (UID: \"e387f3b0-3991-4570-895b-8b59c01c8f36\") " pod="openshift-ingress-canary/ingress-canary-4kbs4"
Oct 31 03:07:08.924972 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.924722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:08.924972 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.924773    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2fzj\" (UniqueName: \"kubernetes.io/projected/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-kube-api-access-n2fzj\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:08.924972 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.924824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/761f94e4-2681-4253-abb6-898d5f67aa98-config-volume\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:08.924972 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.924855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:08.924972 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.924919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9695l\" (UniqueName: \"kubernetes.io/projected/e387f3b0-3991-4570-895b-8b59c01c8f36-kube-api-access-9695l\") pod \"ingress-canary-4kbs4\" (UID: \"e387f3b0-3991-4570-895b-8b59c01c8f36\") " pod="openshift-ingress-canary/ingress-canary-4kbs4"
Oct 31 03:07:08.924972 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.924952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/761f94e4-2681-4253-abb6-898d5f67aa98-metrics-tls\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:08.925220 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.925009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-data-volume\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:08.925220 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.925037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjbgf\" (UniqueName: \"kubernetes.io/projected/761f94e4-2681-4253-abb6-898d5f67aa98-kube-api-access-wjbgf\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:08.925220 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.925109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761f94e4-2681-4253-abb6-898d5f67aa98-tmp-dir\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:08.972159 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.972126    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2bqmj"]
Oct 31 03:07:08.994524 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.994422    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sc79n\""
Oct 31 03:07:08.994524 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.994465    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 03:07:08.994524 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.994516    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 03:07:08.994733 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:08.994527    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wjqqx\""
Oct 31 03:07:09.003547 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.003525    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 03:07:09.025445 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.025416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2fzj\" (UniqueName: \"kubernetes.io/projected/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-kube-api-access-n2fzj\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:09.025573 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.025474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/761f94e4-2681-4253-abb6-898d5f67aa98-config-volume\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:09.025573 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.025498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:09.025573 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.025516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9695l\" (UniqueName: \"kubernetes.io/projected/e387f3b0-3991-4570-895b-8b59c01c8f36-kube-api-access-9695l\") pod \"ingress-canary-4kbs4\" (UID: \"e387f3b0-3991-4570-895b-8b59c01c8f36\") " pod="openshift-ingress-canary/ingress-canary-4kbs4"
Oct 31 03:07:09.025573 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.025532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/761f94e4-2681-4253-abb6-898d5f67aa98-metrics-tls\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:09.025573 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.025553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-data-volume\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:09.025797 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.025574    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjbgf\" (UniqueName: \"kubernetes.io/projected/761f94e4-2681-4253-abb6-898d5f67aa98-kube-api-access-wjbgf\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:09.025797 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.025767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761f94e4-2681-4253-abb6-898d5f67aa98-tmp-dir\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:09.025916 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.025832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-crio-socket\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:09.026033 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.026011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e387f3b0-3991-4570-895b-8b59c01c8f36-cert\") pod \"ingress-canary-4kbs4\" (UID: \"e387f3b0-3991-4570-895b-8b59c01c8f36\") " pod="openshift-ingress-canary/ingress-canary-4kbs4"
Oct 31 03:07:09.026102 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.026029    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-crio-socket\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:09.026102 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.026077    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-data-volume\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:09.026102 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.026076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:09.026237 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.026126    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761f94e4-2681-4253-abb6-898d5f67aa98-tmp-dir\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:09.026325 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.026307    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:09.026379 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.026344    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/761f94e4-2681-4253-abb6-898d5f67aa98-config-volume\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:09.029669 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.029639    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:09.029669 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.029651    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/761f94e4-2681-4253-abb6-898d5f67aa98-metrics-tls\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:09.029842 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.029763    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e387f3b0-3991-4570-895b-8b59c01c8f36-cert\") pod \"ingress-canary-4kbs4\" (UID: \"e387f3b0-3991-4570-895b-8b59c01c8f36\") " pod="openshift-ingress-canary/ingress-canary-4kbs4"
Oct 31 03:07:09.079801 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.079761    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjbgf\" (UniqueName: \"kubernetes.io/projected/761f94e4-2681-4253-abb6-898d5f67aa98-kube-api-access-wjbgf\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:09.079989 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.079867    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2fzj\" (UniqueName: \"kubernetes.io/projected/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-kube-api-access-n2fzj\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:09.079989 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.079921    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9695l\" (UniqueName: \"kubernetes.io/projected/e387f3b0-3991-4570-895b-8b59c01c8f36-kube-api-access-9695l\") pod \"ingress-canary-4kbs4\" (UID: \"e387f3b0-3991-4570-895b-8b59c01c8f36\") " pod="openshift-ingress-canary/ingress-canary-4kbs4"
Oct 31 03:07:09.158239 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.158208    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4kbs4"
Oct 31 03:07:09.166889 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.166861    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:09.171184 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.171156    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:09.589912 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.589712    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4kbs4"]
Oct 31 03:07:09.593025 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.592981    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2bqmj"]
Oct 31 03:07:09.593799 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.593777    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8452g"]
Oct 31 03:07:09.596815 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:07:09.596794    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761f94e4_2681_4253_abb6_898d5f67aa98.slice/crio-0ce09138491b87298dbe7e4fbcf3a24da34384e2c8b444ce6546e00ceb27c792 WatchSource:0}: Error finding container 0ce09138491b87298dbe7e4fbcf3a24da34384e2c8b444ce6546e00ceb27c792: Status 404 returned error can't find the container with id 0ce09138491b87298dbe7e4fbcf3a24da34384e2c8b444ce6546e00ceb27c792
Oct 31 03:07:09.727507 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.727470    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bqmj" event={"ID":"761f94e4-2681-4253-abb6-898d5f67aa98","Type":"ContainerStarted","Data":"0ce09138491b87298dbe7e4fbcf3a24da34384e2c8b444ce6546e00ceb27c792"}
Oct 31 03:07:09.728940 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.728908    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8452g" event={"ID":"c88db770-eb3f-4879-8a41-7e4d7d87fe0e","Type":"ContainerStarted","Data":"7899bf12d44fad7a9eef2c4d4fe5733ff79ec306d7e089ac42d80a249860c292"}
Oct 31 03:07:09.729055 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.728965    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8452g" event={"ID":"c88db770-eb3f-4879-8a41-7e4d7d87fe0e","Type":"ContainerStarted","Data":"d0bc205ffae2f9b64253bd01c30a7391cf6e4fd6e62ed327046110cc77267bd3"}
Oct 31 03:07:09.730007 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:09.729983    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4kbs4" event={"ID":"e387f3b0-3991-4570-895b-8b59c01c8f36","Type":"ContainerStarted","Data":"fc964d0c9ad099f2e03313d5decc2d7cc6de365d8fff1fcc2a1771172aede414"}
Oct 31 03:07:10.512416 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:10.512368    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:07:10.512601 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:10.512380    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:10.512601 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:10.512387    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:07:10.561635 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:10.561602    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 03:07:10.561826 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:10.561723    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z58pm\""
Oct 31 03:07:10.607990 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:10.607946    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ffsgr\""
Oct 31 03:07:10.608427 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:10.608124    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 03:07:10.608427 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:10.608156    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 03:07:10.608427 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:10.608163    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-89z7n\""
Oct 31 03:07:10.608427 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:10.608248    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 03:07:11.737203 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:11.737168    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8452g" event={"ID":"c88db770-eb3f-4879-8a41-7e4d7d87fe0e","Type":"ContainerStarted","Data":"344bb514fb2b8dc69cf2a5652be02781b8c95cc1bdd672935b52bec443933e6f"}
Oct 31 03:07:15.749815 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:15.749721    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bqmj" event={"ID":"761f94e4-2681-4253-abb6-898d5f67aa98","Type":"ContainerStarted","Data":"1b0d683b7fb1061b99be19c412cd0b509e3fc4e61a977d16ecbc171c084bd1f7"}
Oct 31 03:07:15.749815 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:15.749786    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bqmj" event={"ID":"761f94e4-2681-4253-abb6-898d5f67aa98","Type":"ContainerStarted","Data":"633d83a4eadf9ee028975d00f6379cb17b0ef49064222fbbc462282e44faec65"}
Oct 31 03:07:15.750304 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:15.749861    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:15.903465 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:15.903402    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-2bqmj" podStartSLOduration=2.045858529 podStartE2EDuration="7.903379202s" totalImagesPullingTime="5.857520673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:15.902588249 +0000 UTC m=+52.950550437" watchObservedRunningTime="2026-10-31 03:07:15.903379202 +0000 UTC m=+52.951341391"
Oct 31 03:07:20.405311 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:20.405273    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk"]
Oct 31 03:07:20.408595 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:20.408565    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk"
Oct 31 03:07:20.504046 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:20.504011    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6df8z\""
Oct 31 03:07:20.504198 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:20.504097    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 03:07:20.504795 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:20.504775    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk"]
Oct 31 03:07:20.516403 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:20.516365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0077be45-ce2b-46f0-8fe4-e881254f6386-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wtgvk\" (UID: \"0077be45-ce2b-46f0-8fe4-e881254f6386\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk"
Oct 31 03:07:20.617286 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:20.617249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0077be45-ce2b-46f0-8fe4-e881254f6386-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wtgvk\" (UID: \"0077be45-ce2b-46f0-8fe4-e881254f6386\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk"
Oct 31 03:07:20.620852 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:20.620820    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0077be45-ce2b-46f0-8fe4-e881254f6386-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wtgvk\" (UID: \"0077be45-ce2b-46f0-8fe4-e881254f6386\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk"
Oct 31 03:07:20.720521 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:20.720410    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk"
Oct 31 03:07:20.764181 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:20.764140    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8452g" event={"ID":"c88db770-eb3f-4879-8a41-7e4d7d87fe0e","Type":"ContainerStarted","Data":"0dd70b1bd1b61f7cb7df3af312c38da0f265fee249889fcddcb16bb3d4e82b40"}
Oct 31 03:07:20.766401 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:20.766360    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4kbs4" event={"ID":"e387f3b0-3991-4570-895b-8b59c01c8f36","Type":"ContainerStarted","Data":"e12dd502ec93cabd910c897a83e3f66ef48f30c4d3d3ef873f1715d16a15204b"}
Oct 31 03:07:20.873657 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:20.873599    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8452g" podStartSLOduration=2.130337569 podStartE2EDuration="12.873579788s" totalImagesPullingTime="10.743242219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:07:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:20.873301772 +0000 UTC m=+57.921263961" watchObservedRunningTime="2026-10-31 03:07:20.873579788 +0000 UTC m=+57.921541978"
Oct 31 03:07:20.980972 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:20.980852    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4kbs4" podStartSLOduration=2.791876685 podStartE2EDuration="12.980838905s" totalImagesPullingTime="10.18896222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:20.979763643 +0000 UTC m=+58.027725831" watchObservedRunningTime="2026-10-31 03:07:20.980838905 +0000 UTC m=+58.028801092"
Oct 31 03:07:21.082369 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:21.082326    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk"]
Oct 31 03:07:21.086273 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:07:21.086239    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0077be45_ce2b_46f0_8fe4_e881254f6386.slice/crio-ac801bb14592f0e5676ca3681991887116327cdca91a73bbd1ee6e38e91b8bb2 WatchSource:0}: Error finding container ac801bb14592f0e5676ca3681991887116327cdca91a73bbd1ee6e38e91b8bb2: Status 404 returned error can't find the container with id ac801bb14592f0e5676ca3681991887116327cdca91a73bbd1ee6e38e91b8bb2
Oct 31 03:07:21.698309 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:21.698277    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:21.769575 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:21.769537    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk" event={"ID":"0077be45-ce2b-46f0-8fe4-e881254f6386","Type":"ContainerStarted","Data":"ac801bb14592f0e5676ca3681991887116327cdca91a73bbd1ee6e38e91b8bb2"}
Oct 31 03:07:23.777337 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:23.777299    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk" event={"ID":"0077be45-ce2b-46f0-8fe4-e881254f6386","Type":"ContainerStarted","Data":"e2aa0fe2360939746bfc7521eaed3ce4baf481f776f96cbce9d4ae15a6faab17"}
Oct 31 03:07:23.777765 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:23.777505    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk"
Oct 31 03:07:23.782244 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:23.782221    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk"
Oct 31 03:07:23.884457 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:23.884389    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk" podStartSLOduration=1.849043775 podStartE2EDuration="3.884372957s" totalImagesPullingTime="2.035329182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:23.884295191 +0000 UTC m=+60.932257379" watchObservedRunningTime="2026-10-31 03:07:23.884372957 +0000 UTC m=+60.932335146"
Oct 31 03:07:25.037738 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.037704    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-7rr4c"]
Oct 31 03:07:25.040198 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.040182    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:07:25.084238 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.084196    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"]
Oct 31 03:07:25.086586 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.086563    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.129901 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.129855    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 03:07:25.130089 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.129926    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 03:07:25.130089 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.129943    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-79t2s\""
Oct 31 03:07:25.139080 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.139050    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 03:07:25.142979 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.142942    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7rr4c"]
Oct 31 03:07:25.150227 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.150199    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"]
Oct 31 03:07:25.153322 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.153295    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cb392b3-c5a6-4ae1-af06-82f167ed3c27-tmp\") pod \"downloads-6dd789c8b7-7rr4c\" (UID: \"6cb392b3-c5a6-4ae1-af06-82f167ed3c27\") " pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:07:25.153467 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.153394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5j7c\" (UniqueName: \"kubernetes.io/projected/6cb392b3-c5a6-4ae1-af06-82f167ed3c27-kube-api-access-k5j7c\") pod \"downloads-6dd789c8b7-7rr4c\" (UID: \"6cb392b3-c5a6-4ae1-af06-82f167ed3c27\") " pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:07:25.198247 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.198217    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 03:07:25.198425 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.198334    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 03:07:25.198425 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.198370    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 03:07:25.198425 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.198379    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-87zff\""
Oct 31 03:07:25.198425 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.198401    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 03:07:25.254472 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.254431    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5j7c\" (UniqueName: \"kubernetes.io/projected/6cb392b3-c5a6-4ae1-af06-82f167ed3c27-kube-api-access-k5j7c\") pod \"downloads-6dd789c8b7-7rr4c\" (UID: \"6cb392b3-c5a6-4ae1-af06-82f167ed3c27\") " pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:07:25.254472 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.254474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.254472 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.254497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.254753 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.254565    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.254753 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.254697    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22m44\" (UniqueName: \"kubernetes.io/projected/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-kube-api-access-22m44\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.254753 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.254725    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cb392b3-c5a6-4ae1-af06-82f167ed3c27-tmp\") pod \"downloads-6dd789c8b7-7rr4c\" (UID: \"6cb392b3-c5a6-4ae1-af06-82f167ed3c27\") " pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:07:25.255167 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.255144    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cb392b3-c5a6-4ae1-af06-82f167ed3c27-tmp\") pod \"downloads-6dd789c8b7-7rr4c\" (UID: \"6cb392b3-c5a6-4ae1-af06-82f167ed3c27\") " pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:07:25.308689 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.308614    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5j7c\" (UniqueName: \"kubernetes.io/projected/6cb392b3-c5a6-4ae1-af06-82f167ed3c27-kube-api-access-k5j7c\") pod \"downloads-6dd789c8b7-7rr4c\" (UID: \"6cb392b3-c5a6-4ae1-af06-82f167ed3c27\") " pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:07:25.350309 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.350264    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:07:25.355680 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.355651    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.355777 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.355695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.355857 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.355835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.355932 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.355899    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22m44\" (UniqueName: \"kubernetes.io/projected/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-kube-api-access-22m44\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.356484 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.356454    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.358255 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.358234    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.358354 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.358330    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.410475 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.410399    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22m44\" (UniqueName: \"kubernetes.io/projected/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-kube-api-access-22m44\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.698387 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.698348    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:25.755839 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.755812    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:25.758658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.758634    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7rr4c"]
Oct 31 03:07:25.763590 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:07:25.763564    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb392b3_c5a6_4ae1_af06_82f167ed3c27.slice/crio-7b45609908eb3f178730eee8e8d374b1e4a030142d4c973e9613e574c2d5bd1a WatchSource:0}: Error finding container 7b45609908eb3f178730eee8e8d374b1e4a030142d4c973e9613e574c2d5bd1a: Status 404 returned error can't find the container with id 7b45609908eb3f178730eee8e8d374b1e4a030142d4c973e9613e574c2d5bd1a
Oct 31 03:07:25.784383 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:25.784346    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7rr4c" event={"ID":"6cb392b3-c5a6-4ae1-af06-82f167ed3c27","Type":"ContainerStarted","Data":"7b45609908eb3f178730eee8e8d374b1e4a030142d4c973e9613e574c2d5bd1a"}
Oct 31 03:07:26.067686 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:26.067609    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"]
Oct 31 03:07:26.071353 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:07:26.071324    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16799a99_285f_40d5_acbb_a8fbe0d7a2d9.slice/crio-6584704ea7674d07cd35fed8ec485303303ce0e50a7fefa74cc717303f48f534 WatchSource:0}: Error finding container 6584704ea7674d07cd35fed8ec485303303ce0e50a7fefa74cc717303f48f534: Status 404 returned error can't find the container with id 6584704ea7674d07cd35fed8ec485303303ce0e50a7fefa74cc717303f48f534
Oct 31 03:07:26.793116 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:26.793077    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq" event={"ID":"16799a99-285f-40d5-acbb-a8fbe0d7a2d9","Type":"ContainerStarted","Data":"6584704ea7674d07cd35fed8ec485303303ce0e50a7fefa74cc717303f48f534"}
Oct 31 03:07:28.799966 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:28.799930    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq" event={"ID":"16799a99-285f-40d5-acbb-a8fbe0d7a2d9","Type":"ContainerStarted","Data":"a326d2c0511ca52801c544fda42e3d0a4f40a141509f7c224f97dc6ed77fba4a"}
Oct 31 03:07:28.800457 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:28.799992    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq" event={"ID":"16799a99-285f-40d5-acbb-a8fbe0d7a2d9","Type":"ContainerStarted","Data":"8d1c13ee352b96cf3e2f6280dd1d650a555f47c67c460c7d77ae9c3719d485a2"}
Oct 31 03:07:28.938161 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:28.938049    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq" podStartSLOduration=1.383465271 podStartE2EDuration="3.938034256s" totalImagesPullingTime="2.554568985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:28.937279891 +0000 UTC m=+65.985242079" watchObservedRunningTime="2026-10-31 03:07:28.938034256 +0000 UTC m=+65.985996443"
Oct 31 03:07:29.389786 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.389732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:07:29.390037 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.389837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:29.484174 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.484130    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 03:07:29.484174 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.484165    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 03:07:29.490922 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.490501    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qnj\" (UniqueName: \"kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj\") pod \"network-check-target-mr7hm\" (UID: \"ea79f848-e58a-463b-8c7f-d6f6dd0125b3\") " pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:07:29.492687 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.492667    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:07:29.493011 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.492996    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:29.538645 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.538609    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 03:07:29.589143 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.589111    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 03:07:29.594701 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.594676    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94qnj\" (UniqueName: \"kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj\") pod \"network-check-target-mr7hm\" (UID: \"ea79f848-e58a-463b-8c7f-d6f6dd0125b3\") " pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:07:29.773804 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.773723    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ffsgr\""
Oct 31 03:07:29.776818 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.776795    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:07:29.777819 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.777799    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z58pm\""
Oct 31 03:07:29.781022 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.781000    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:29.782689 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.782669    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-89z7n\""
Oct 31 03:07:29.786056 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:29.786027    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:07:30.145228 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:30.145190    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sl5hn"]
Oct 31 03:07:30.148631 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:07:30.148598    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbec67c_68ae_48f9_b780_bff7931da269.slice/crio-0fbf76a6d280d81147161a5c521406c20a8b3f4feeb53eb9d38839dacf7e4040 WatchSource:0}: Error finding container 0fbf76a6d280d81147161a5c521406c20a8b3f4feeb53eb9d38839dacf7e4040: Status 404 returned error can't find the container with id 0fbf76a6d280d81147161a5c521406c20a8b3f4feeb53eb9d38839dacf7e4040
Oct 31 03:07:30.192021 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:30.191988    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mr7hm"]
Oct 31 03:07:30.195009 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:30.194944    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ldcbk"]
Oct 31 03:07:30.195576 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:07:30.195545    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea79f848_e58a_463b_8c7f_d6f6dd0125b3.slice/crio-7d1c1147925b96371116ba5e77a174d636ff3c84082a86c26a4f49ce8f9317e2 WatchSource:0}: Error finding container 7d1c1147925b96371116ba5e77a174d636ff3c84082a86c26a4f49ce8f9317e2: Status 404 returned error can't find the container with id 7d1c1147925b96371116ba5e77a174d636ff3c84082a86c26a4f49ce8f9317e2
Oct 31 03:07:30.198109 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:07:30.198089    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35c8150_ce21_4660_ad5d_e5cc1fce684b.slice/crio-0eb0e92f85052d88c64727b4db314b5a5875543c670e6adec9799e6448e3edf4 WatchSource:0}: Error finding container 0eb0e92f85052d88c64727b4db314b5a5875543c670e6adec9799e6448e3edf4: Status 404 returned error can't find the container with id 0eb0e92f85052d88c64727b4db314b5a5875543c670e6adec9799e6448e3edf4
Oct 31 03:07:30.809238 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:30.809200    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ldcbk" event={"ID":"d35c8150-ce21-4660-ad5d-e5cc1fce684b","Type":"ContainerStarted","Data":"0eb0e92f85052d88c64727b4db314b5a5875543c670e6adec9799e6448e3edf4"}
Oct 31 03:07:30.810532 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:30.810501    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sl5hn" event={"ID":"0fbec67c-68ae-48f9-b780-bff7931da269","Type":"ContainerStarted","Data":"0fbf76a6d280d81147161a5c521406c20a8b3f4feeb53eb9d38839dacf7e4040"}
Oct 31 03:07:30.811767 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:30.811734    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mr7hm" event={"ID":"ea79f848-e58a-463b-8c7f-d6f6dd0125b3","Type":"ContainerStarted","Data":"7d1c1147925b96371116ba5e77a174d636ff3c84082a86c26a4f49ce8f9317e2"}
Oct 31 03:07:31.618942 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.617714    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"]
Oct 31 03:07:31.620982 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.620373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:31.670194 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.670156    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 03:07:31.715517 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.715485    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lfdkn\""
Oct 31 03:07:31.715989 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.715971    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 03:07:31.717452 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.717393    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"]
Oct 31 03:07:31.736903 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.733629    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2cjh2"]
Oct 31 03:07:31.743802 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.742111    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:31.765619 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.765584    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"]
Oct 31 03:07:31.768220 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.768194    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:31.795845 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.795809    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"]
Oct 31 03:07:31.796837 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.796816    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 03:07:31.797008 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.796859    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 03:07:31.797008 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.796951    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 03:07:31.797083 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.797028    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k74vl\""
Oct 31 03:07:31.809439 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.809407    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2941755b-c963-4514-a42a-c918a8283273-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:31.809618 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.809447    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2941755b-c963-4514-a42a-c918a8283273-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:31.809618 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.809479    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m5bp\" (UniqueName: \"kubernetes.io/projected/2941755b-c963-4514-a42a-c918a8283273-kube-api-access-9m5bp\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:31.809618 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.809512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2941755b-c963-4514-a42a-c918a8283273-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:31.854242 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.854029    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 03:07:31.854242 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.854197    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 03:07:31.854620 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.854568    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4744q\""
Oct 31 03:07:31.897147 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.897069    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910331    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-textfile\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/da297bce-ca72-4dea-b02c-bcda6f401ebe-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2941755b-c963-4514-a42a-c918a8283273-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-sys\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910565    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvlpv\" (UniqueName: \"kubernetes.io/projected/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-api-access-xvlpv\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m5bp\" (UniqueName: \"kubernetes.io/projected/2941755b-c963-4514-a42a-c918a8283273-kube-api-access-9m5bp\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-tls\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-wtmp\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910705    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177793bf-7bb1-4b0b-adb6-ba40483ae237-metrics-client-ca\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4q5r\" (UniqueName: \"kubernetes.io/projected/177793bf-7bb1-4b0b-adb6-ba40483ae237-kube-api-access-b4q5r\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:31.912604 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da297bce-ca72-4dea-b02c-bcda6f401ebe-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:31.915020 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2941755b-c963-4514-a42a-c918a8283273-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:31.915020 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910923    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:31.915020 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2941755b-c963-4514-a42a-c918a8283273-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:31.915020 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.910976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-root\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:31.915020 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.914233    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2941755b-c963-4514-a42a-c918a8283273-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:31.915898 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.915830    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2941755b-c963-4514-a42a-c918a8283273-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:31.917085 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.917056    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2941755b-c963-4514-a42a-c918a8283273-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:31.969026 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:31.968948    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m5bp\" (UniqueName: \"kubernetes.io/projected/2941755b-c963-4514-a42a-c918a8283273-kube-api-access-9m5bp\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012424    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177793bf-7bb1-4b0b-adb6-ba40483ae237-metrics-client-ca\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4q5r\" (UniqueName: \"kubernetes.io/projected/177793bf-7bb1-4b0b-adb6-ba40483ae237-kube-api-access-b4q5r\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da297bce-ca72-4dea-b02c-bcda6f401ebe-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-root\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012575    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-textfile\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/da297bce-ca72-4dea-b02c-bcda6f401ebe-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-sys\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlpv\" (UniqueName: \"kubernetes.io/projected/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-api-access-xvlpv\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012763    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-tls\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012793    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-wtmp\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.013314 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.012817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:32.014098 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.013983    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-sys\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.014838 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.014540    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-textfile\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.014838 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.014552    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/da297bce-ca72-4dea-b02c-bcda6f401ebe-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:32.014838 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.014577    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.014838 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.014650    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177793bf-7bb1-4b0b-adb6-ba40483ae237-metrics-client-ca\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.014838 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.014655    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-root\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.014838 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.014813    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-wtmp\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.015138 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.015113    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:32.015421 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.015397    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da297bce-ca72-4dea-b02c-bcda6f401ebe-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:32.017418 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.017359    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:32.017931 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.017905    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.018023 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.017927    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:32.022992 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.022962    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-tls\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.070757 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.070725    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4q5r\" (UniqueName: \"kubernetes.io/projected/177793bf-7bb1-4b0b-adb6-ba40483ae237-kube-api-access-b4q5r\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.071087 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.071030    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlpv\" (UniqueName: \"kubernetes.io/projected/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-api-access-xvlpv\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:32.085782 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.085749    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:32.239863 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.239633    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:32.361775 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.361736    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:32.374056 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:07:32.374021    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177793bf_7bb1_4b0b_adb6_ba40483ae237.slice/crio-fd7c99e40f4ebc498610ab29dfdb8dc6e44d42a8645542070173227f0ad8d6c0 WatchSource:0}: Error finding container fd7c99e40f4ebc498610ab29dfdb8dc6e44d42a8645542070173227f0ad8d6c0: Status 404 returned error can't find the container with id fd7c99e40f4ebc498610ab29dfdb8dc6e44d42a8645542070173227f0ad8d6c0
Oct 31 03:07:32.534460 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.534385    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"]
Oct 31 03:07:32.538747 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:07:32.538705    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda297bce_ca72_4dea_b02c_bcda6f401ebe.slice/crio-ef787f9a5262b613175feccfeb02e9b6716696ed5ef06b5dd0299392ac5f5fff WatchSource:0}: Error finding container ef787f9a5262b613175feccfeb02e9b6716696ed5ef06b5dd0299392ac5f5fff: Status 404 returned error can't find the container with id ef787f9a5262b613175feccfeb02e9b6716696ed5ef06b5dd0299392ac5f5fff
Oct 31 03:07:32.629936 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.629900    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"]
Oct 31 03:07:32.633622 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:07:32.633589    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2941755b_c963_4514_a42a_c918a8283273.slice/crio-1088bb71c93be618fad92d4b06d39e972f2693be186ddc175d1f790569d4d0bb WatchSource:0}: Error finding container 1088bb71c93be618fad92d4b06d39e972f2693be186ddc175d1f790569d4d0bb: Status 404 returned error can't find the container with id 1088bb71c93be618fad92d4b06d39e972f2693be186ddc175d1f790569d4d0bb
Oct 31 03:07:32.822488 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.822417    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58" event={"ID":"da297bce-ca72-4dea-b02c-bcda6f401ebe","Type":"ContainerStarted","Data":"ef787f9a5262b613175feccfeb02e9b6716696ed5ef06b5dd0299392ac5f5fff"}
Oct 31 03:07:32.824158 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.824122    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cjh2" event={"ID":"177793bf-7bb1-4b0b-adb6-ba40483ae237","Type":"ContainerStarted","Data":"fd7c99e40f4ebc498610ab29dfdb8dc6e44d42a8645542070173227f0ad8d6c0"}
Oct 31 03:07:32.828211 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.828174    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sl5hn" event={"ID":"0fbec67c-68ae-48f9-b780-bff7931da269","Type":"ContainerStarted","Data":"bda02bd6f79467d5ae711d0739ecd622b9f0a7fea4df25a87fb48ee32ca7382a"}
Oct 31 03:07:32.828403 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.828240    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sl5hn" event={"ID":"0fbec67c-68ae-48f9-b780-bff7931da269","Type":"ContainerStarted","Data":"fe38e9e116db7db4fd4aa2c408d4448c4c5357d6bb37384bf86452ce79b7b8ae"}
Oct 31 03:07:32.831701 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.831659    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr" event={"ID":"2941755b-c963-4514-a42a-c918a8283273","Type":"ContainerStarted","Data":"2ccceb47186dd3d4184e85cc2895ad1f838913bb86a679b7c880ff592e6e3b43"}
Oct 31 03:07:32.831848 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.831714    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr" event={"ID":"2941755b-c963-4514-a42a-c918a8283273","Type":"ContainerStarted","Data":"5daa9de73c99d9ca1d0848acedc0885a15bfe12fd5a0a5d8927887a37af27951"}
Oct 31 03:07:32.831848 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.831739    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr" event={"ID":"2941755b-c963-4514-a42a-c918a8283273","Type":"ContainerStarted","Data":"1088bb71c93be618fad92d4b06d39e972f2693be186ddc175d1f790569d4d0bb"}
Oct 31 03:07:32.949519 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:32.948559    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sl5hn" podStartSLOduration=67.042858464 podStartE2EDuration="1m8.948539361s" totalImagesPullingTime="1.905680897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:32.946939584 +0000 UTC m=+69.994901771" watchObservedRunningTime="2026-10-31 03:07:32.948539361 +0000 UTC m=+69.996501550"
Oct 31 03:07:33.368632 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.368586    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-74c979ccc9-r9gs5"]
Oct 31 03:07:33.371240 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.371213    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.421365 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.421330    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fb2gc\""
Oct 31 03:07:33.422671 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.422179    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 03:07:33.422671 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.422286    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 03:07:33.422671 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.422384    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 03:07:33.422671 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.422502    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 03:07:33.422671 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.422589    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 03:07:33.428996 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.428917    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c979ccc9-r9gs5"]
Oct 31 03:07:33.528485 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.528439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-tmp\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.528658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.528509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-service-ca\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.528658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.528538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zvxt\" (UniqueName: \"kubernetes.io/projected/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-kube-api-access-6zvxt\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.528658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.528577    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-oauth-serving-cert\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.528658 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.528638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-serving-cert\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.528992 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.528678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-config\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.528992 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.528710    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-oauth-config\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.630148 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.630061    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-serving-cert\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.630148 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.630120    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-config\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.630672 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.630149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-oauth-config\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.630672 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.630176    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-tmp\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.630672 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.630244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-service-ca\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.630672 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.630271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zvxt\" (UniqueName: \"kubernetes.io/projected/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-kube-api-access-6zvxt\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.630672 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.630311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-oauth-serving-cert\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.631212 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.631142    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-oauth-serving-cert\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.631523 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.631473    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-service-ca\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.631850 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.631829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-tmp\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.633017 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.632414    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-config\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.640042 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.639984    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-oauth-config\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.642522 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.642486    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-serving-cert\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.701097 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.701058    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zvxt\" (UniqueName: \"kubernetes.io/projected/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-kube-api-access-6zvxt\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:33.986255 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:33.986167    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:34.405223 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:34.405178    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c979ccc9-r9gs5"]
Oct 31 03:07:34.451130 ip-10-0-139-164 kubenswrapper[2576]: W1031 03:07:34.444946    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b43c0c_e9fc_4d01_8034_2c1d10c3c684.slice/crio-5a19334855370632f158cab70d5975524cf2ce6394dcc4c55839ea58a91013d4 WatchSource:0}: Error finding container 5a19334855370632f158cab70d5975524cf2ce6394dcc4c55839ea58a91013d4: Status 404 returned error can't find the container with id 5a19334855370632f158cab70d5975524cf2ce6394dcc4c55839ea58a91013d4
Oct 31 03:07:34.851468 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:34.851421    2576 generic.go:356] "Generic (PLEG): container finished" podID="177793bf-7bb1-4b0b-adb6-ba40483ae237" containerID="3c6eeb53a30fcd8f831d7764f8856a2f49cb8d5647d339ddc0480ee7caca66df" exitCode=0
Oct 31 03:07:34.851961 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:34.851537    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cjh2" event={"ID":"177793bf-7bb1-4b0b-adb6-ba40483ae237","Type":"ContainerDied","Data":"3c6eeb53a30fcd8f831d7764f8856a2f49cb8d5647d339ddc0480ee7caca66df"}
Oct 31 03:07:34.857132 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:34.857058    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c979ccc9-r9gs5" event={"ID":"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684","Type":"ContainerStarted","Data":"5a19334855370632f158cab70d5975524cf2ce6394dcc4c55839ea58a91013d4"}
Oct 31 03:07:35.878516 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:35.871696    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58" event={"ID":"da297bce-ca72-4dea-b02c-bcda6f401ebe","Type":"ContainerStarted","Data":"1022926be76875b6f27f87d4c87cb33bfdb9a974c9090cccd9f8ef4c39af4aa9"}
Oct 31 03:07:35.878516 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:35.871899    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58" event={"ID":"da297bce-ca72-4dea-b02c-bcda6f401ebe","Type":"ContainerStarted","Data":"57b0f325c824a0c1ecab0c13a221d7befe0c9a8aad04cf7ca734a8fdee852454"}
Oct 31 03:07:35.878516 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:35.871987    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58" event={"ID":"da297bce-ca72-4dea-b02c-bcda6f401ebe","Type":"ContainerStarted","Data":"1a990c2661dbe2f4dd9059c1803f23beadf780a9ef97aa27336fff640ede9ae4"}
Oct 31 03:07:35.878516 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:35.876094    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cjh2" event={"ID":"177793bf-7bb1-4b0b-adb6-ba40483ae237","Type":"ContainerStarted","Data":"bb99e70624809a376c0bfc49833b638fb72d60ae6003f39812d93abf91d7cac4"}
Oct 31 03:07:35.878516 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:35.876137    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cjh2" event={"ID":"177793bf-7bb1-4b0b-adb6-ba40483ae237","Type":"ContainerStarted","Data":"ba38fb0a43ad94604ca165bea792ce0041aa7685253d6015caaee5a5cdf8abcf"}
Oct 31 03:07:35.880193 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:35.880153    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr" event={"ID":"2941755b-c963-4514-a42a-c918a8283273","Type":"ContainerStarted","Data":"d71c37bcb9a996ae5bd825b3f0c7b757fc111f419fc6fd2d099efc4735446df8"}
Oct 31 03:07:36.111463 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.111356    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58" podStartSLOduration=2.413634817 podStartE2EDuration="5.111338188s" totalImagesPullingTime="2.697703371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:35.985471579 +0000 UTC m=+73.033433773" watchObservedRunningTime="2026-10-31 03:07:36.111338188 +0000 UTC m=+73.159300378"
Oct 31 03:07:36.111749 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.111653    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr" podStartSLOduration=2.927256143 podStartE2EDuration="5.111643752s" totalImagesPullingTime="2.184387609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:36.110817976 +0000 UTC m=+73.158780166" watchObservedRunningTime="2026-10-31 03:07:36.111643752 +0000 UTC m=+73.159605942"
Oct 31 03:07:36.359857 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.359104    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2cjh2" podStartSLOduration=3.114310437 podStartE2EDuration="5.359082951s" totalImagesPullingTime="2.156454003s" totalInitContainerRuntime="88.318511ms" isStatefulPod=true podCreationTimestamp="2026-10-31 03:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:36.223446483 +0000 UTC m=+73.271408672" watchObservedRunningTime="2026-10-31 03:07:36.359082951 +0000 UTC m=+73.407045140"
Oct 31 03:07:36.360136 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.360033    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"]
Oct 31 03:07:36.363299 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.363231    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:36.407419 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.407355    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4kbs4_e387f3b0-3991-4570-895b-8b59c01c8f36/serve-healthcheck-canary/0.log"
Oct 31 03:07:36.417446 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.417410    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-57bwv\""
Oct 31 03:07:36.417710 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.417530    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 03:07:36.425391 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.425347    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"]
Oct 31 03:07:36.462806 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.462595    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8bd5\" (UniqueName: \"kubernetes.io/projected/a18739ba-0914-4b52-bcda-4a5fafe22225-kube-api-access-g8bd5\") pod \"monitoring-plugin-77f4f76594-p2kxj\" (UID: \"a18739ba-0914-4b52-bcda-4a5fafe22225\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:36.462806 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.462668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a18739ba-0914-4b52-bcda-4a5fafe22225-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p2kxj\" (UID: \"a18739ba-0914-4b52-bcda-4a5fafe22225\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:36.563436 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.563395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a18739ba-0914-4b52-bcda-4a5fafe22225-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p2kxj\" (UID: \"a18739ba-0914-4b52-bcda-4a5fafe22225\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:36.563637 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.563575    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8bd5\" (UniqueName: \"kubernetes.io/projected/a18739ba-0914-4b52-bcda-4a5fafe22225-kube-api-access-g8bd5\") pod \"monitoring-plugin-77f4f76594-p2kxj\" (UID: \"a18739ba-0914-4b52-bcda-4a5fafe22225\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:36.567201 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.567171    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a18739ba-0914-4b52-bcda-4a5fafe22225-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p2kxj\" (UID: \"a18739ba-0914-4b52-bcda-4a5fafe22225\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:36.618072 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.617994    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8bd5\" (UniqueName: \"kubernetes.io/projected/a18739ba-0914-4b52-bcda-4a5fafe22225-kube-api-access-g8bd5\") pod \"monitoring-plugin-77f4f76594-p2kxj\" (UID: \"a18739ba-0914-4b52-bcda-4a5fafe22225\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:36.673349 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.673310    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7887dcbc55-2plp6"]
Oct 31 03:07:36.676301 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.676274    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.680328 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.680298    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:36.725514 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.725483    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dzlls\""
Oct 31 03:07:36.725683 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.725629    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 03:07:36.725683 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.725661    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 03:07:36.725769 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.725697    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5quaqkm7oduoe\""
Oct 31 03:07:36.725769 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.725729    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 03:07:36.725913 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.725856    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 03:07:36.733316 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.733285    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7887dcbc55-2plp6"]
Oct 31 03:07:36.764907 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.764700    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a4d955b0-1418-499f-9036-961443f1e130-audit-log\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.764907 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.764762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n965n\" (UniqueName: \"kubernetes.io/projected/a4d955b0-1418-499f-9036-961443f1e130-kube-api-access-n965n\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.764907 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.764789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-client-ca-bundle\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.764907 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.764817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4d955b0-1418-499f-9036-961443f1e130-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.765213 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.765005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-secret-metrics-server-client-certs\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.765213 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.765040    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-secret-metrics-server-tls\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.765213 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.765071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a4d955b0-1418-499f-9036-961443f1e130-metrics-server-audit-profiles\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.868952 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.866297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a4d955b0-1418-499f-9036-961443f1e130-audit-log\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.868952 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.866362    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n965n\" (UniqueName: \"kubernetes.io/projected/a4d955b0-1418-499f-9036-961443f1e130-kube-api-access-n965n\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.868952 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.866394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-client-ca-bundle\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.868952 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.866421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4d955b0-1418-499f-9036-961443f1e130-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.868952 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.866501    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-secret-metrics-server-client-certs\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.868952 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.866532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-secret-metrics-server-tls\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.868952 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.866566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a4d955b0-1418-499f-9036-961443f1e130-metrics-server-audit-profiles\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.868952 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.867695    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a4d955b0-1418-499f-9036-961443f1e130-metrics-server-audit-profiles\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.868952 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.867976    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a4d955b0-1418-499f-9036-961443f1e130-audit-log\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.870466 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.870326    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4d955b0-1418-499f-9036-961443f1e130-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.872664 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.872632    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-client-ca-bundle\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.872766 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.872678    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-secret-metrics-server-tls\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.875984 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.875961    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-secret-metrics-server-client-certs\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.896964 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.896835    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mr7hm" event={"ID":"ea79f848-e58a-463b-8c7f-d6f6dd0125b3","Type":"ContainerStarted","Data":"1acbfdaedf6006104fd6797a82943e9b9557642025a278e162ccd76ad4f3bd93"}
Oct 31 03:07:36.897944 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.897529    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:07:36.922650 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.922458    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n965n\" (UniqueName: \"kubernetes.io/projected/a4d955b0-1418-499f-9036-961443f1e130-kube-api-access-n965n\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:36.992050 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:36.992013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:37.005687 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:37.005621    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mr7hm" podStartSLOduration=67.339756085 podStartE2EDuration="1m13.005599985s" totalImagesPullingTime="5.6658439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:37.004720187 +0000 UTC m=+74.052682375" watchObservedRunningTime="2026-10-31 03:07:37.005599985 +0000 UTC m=+74.053562174"
Oct 31 03:07:37.083513 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:37.083454    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"]
Oct 31 03:07:37.384791 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:37.384492    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7887dcbc55-2plp6"]
Oct 31 03:07:37.901857 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:37.901781    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6" event={"ID":"a4d955b0-1418-499f-9036-961443f1e130","Type":"ContainerStarted","Data":"4e053babb6df547e91d28ea281ca9956a3bea569d86749e30b2b49dca2c973f3"}
Oct 31 03:07:37.903981 ip-10-0-139-164 kubenswrapper[2576]: I1031 03:07:37.903948    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj" event={"ID":"a18739ba-0914-4b52-bcda-4a5fafe22225","Type":"ContainerStarted","Data":"a1dffbd195d3834921291b32d6ee069053f5d1df3bdad560e047c130664e132a"}
Oct 31 03:07:38.365291 ip-10-0-139-164 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 03:07:38.378671 ip-10-0-139-164 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 03:07:38.378945 ip-10-0-139-164 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 03:07:38.379253 ip-10-0-139-164 systemd[1]: kubelet.service: Consumed 4.240s CPU time.
Oct 31 03:07:38.386797 ip-10-0-139-164 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 03:07:38.479273 ip-10-0-139-164 kubenswrapper[5415]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:07:38.479273 ip-10-0-139-164 kubenswrapper[5415]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 03:07:38.479273 ip-10-0-139-164 kubenswrapper[5415]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:07:38.479273 ip-10-0-139-164 kubenswrapper[5415]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:07:38.482946 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482805    5415 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:38.482946 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482941    5415 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:38.482946 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482947    5415 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:38.482946 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482952    5415 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:38.482946 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482956    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482961    5415 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482965    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482970    5415 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482974    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482980    5415 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482984    5415 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482989    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482993    5415 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.482997    5415 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483010    5415 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483015    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483020    5415 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483024    5415 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483028    5415 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483032    5415 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483037    5415 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483041    5415 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483045    5415 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:38.483195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483049    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483053    5415 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483058    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483063    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483068    5415 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483072    5415 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483076    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483080    5415 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483085    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483089    5415 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483093    5415 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483098    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483102    5415 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483106    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483112    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483116    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483121    5415 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483126    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483130    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:38.483710 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483134    5415 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483138    5415 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483142    5415 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483146    5415 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483151    5415 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483156    5415 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483161    5415 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483165    5415 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483169    5415 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483173    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483181    5415 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483188    5415 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483193    5415 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483197    5415 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483201    5415 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483207    5415 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483211    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483216    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483220    5415 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483224    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:38.484257 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483228    5415 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483233    5415 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483237    5415 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483241    5415 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483245    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483249    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483254    5415 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483258    5415 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483262    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483267    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483271    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483275    5415 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483279    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483284    5415 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483288    5415 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483292    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483296    5415 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483300    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483305    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483310    5415 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:38.484809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483315    5415 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483319    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483323    5415 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483327    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483332    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483337    5415 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483341    5415 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483346    5415 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483350    5415 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483354    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483358    5415 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483365    5415 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483371    5415 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483376    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483380    5415 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483384    5415 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483389    5415 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483393    5415 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483400    5415 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:38.485336 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483404    5415 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483408    5415 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483412    5415 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483416    5415 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.483421    5415 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484768    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484777    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484782    5415 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484787    5415 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484791    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484796    5415 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484800    5415 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484806    5415 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484810    5415 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484814    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484819    5415 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484823    5415 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484827    5415 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484834    5415 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:38.485910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484840    5415 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484845    5415 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484849    5415 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484853    5415 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484858    5415 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484862    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484867    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484872    5415 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484890    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484894    5415 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484902    5415 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484910    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484915    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484920    5415 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484925    5415 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484931    5415 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484936    5415 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484940    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484945    5415 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:38.486437 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484949    5415 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484953    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484957    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484961    5415 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484966    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484969    5415 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484974    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484979    5415 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484983    5415 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484988    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484992    5415 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.484996    5415 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485000    5415 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485004    5415 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485008    5415 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485012    5415 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485016    5415 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485020    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485025    5415 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:38.487028 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485029    5415 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485033    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485037    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485041    5415 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485047    5415 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485052    5415 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485056    5415 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485060    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485065    5415 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485069    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485073    5415 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485077    5415 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485081    5415 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485086    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485090    5415 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485094    5415 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485098    5415 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485102    5415 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485106    5415 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485111    5415 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:38.487530 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485115    5415 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485120    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485125    5415 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485129    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485133    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485138    5415 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485142    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485146    5415 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485150    5415 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485155    5415 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485159    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485163    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485167    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485171    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485175    5415 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485179    5415 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485184    5415 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485188    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485192    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485197    5415 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:38.488146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485201    5415 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485205    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485209    5415 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485213    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485217    5415 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485222    5415 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485226    5415 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485230    5415 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485234    5415 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485238    5415 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485242    5415 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485247    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485250    5415 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.485254    5415 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486157    5415 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486165    5415 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486170    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486174    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486179    5415 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486183    5415 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:38.488743 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486187    5415 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486191    5415 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486195    5415 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486199    5415 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486204    5415 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486210    5415 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486216    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486222    5415 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486226    5415 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486230    5415 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486235    5415 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486239    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486243    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486247    5415 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486252    5415 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486256    5415 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486260    5415 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486264    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486268    5415 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:38.489352 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486272    5415 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486277    5415 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486281    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486285    5415 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486290    5415 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486294    5415 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486298    5415 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486303    5415 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486307    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486311    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486315    5415 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486319    5415 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486322    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486327    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486331    5415 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486334    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486340    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486344    5415 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486348    5415 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486353    5415 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:38.489980 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486357    5415 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486361    5415 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486365    5415 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486369    5415 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486373    5415 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486378    5415 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486382    5415 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486386    5415 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486390    5415 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486394    5415 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486398    5415 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486402    5415 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486406    5415 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486411    5415 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486415    5415 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486419    5415 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486423    5415 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486428    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486432    5415 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486437    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:38.490608 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486441    5415 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486446    5415 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486450    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486455    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486460    5415 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486464    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486468    5415 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486472    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486476    5415 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486481    5415 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486485    5415 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486490    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486494    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486498    5415 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486502    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486506    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486510    5415 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486514    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486519    5415 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:38.491317 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486523    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486527    5415 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486531    5415 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486535    5415 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486539    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486543    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486548    5415 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486552    5415 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486556    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486560    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486564    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486572    5415 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486578    5415 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486584    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486589    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486593    5415 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486597    5415 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486601    5415 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486605    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:38.491955 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486609    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486613    5415 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.486617    5415 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486758    5415 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486773    5415 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486783    5415 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486789    5415 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486797    5415 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486805    5415 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486813    5415 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486821    5415 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486826    5415 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486831    5415 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486837    5415 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486843    5415 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486848    5415 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486853    5415 flags.go:64] FLAG: --cgroup-root=""
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486858    5415 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486863    5415 flags.go:64] FLAG: --client-ca-file=""
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486868    5415 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486888    5415 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486896    5415 flags.go:64] FLAG: --cluster-domain=""
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486901    5415 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 03:07:38.492651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486906    5415 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486911    5415 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486917    5415 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486924    5415 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486929    5415 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486936    5415 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486941    5415 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486946    5415 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486951    5415 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486956    5415 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486961    5415 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486966    5415 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486972    5415 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486977    5415 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486982    5415 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486986    5415 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.486994    5415 flags.go:64] FLAG: --enable-server="true"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487004    5415 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487011    5415 flags.go:64] FLAG: --event-burst="100"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487016    5415 flags.go:64] FLAG: --event-qps="50"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487021    5415 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487025    5415 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487031    5415 flags.go:64] FLAG: --eviction-hard=""
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487038    5415 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 03:07:38.493529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487043    5415 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487048    5415 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487053    5415 flags.go:64] FLAG: --eviction-soft=""
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487058    5415 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487063    5415 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487068    5415 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487073    5415 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487078    5415 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487083    5415 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487088    5415 flags.go:64] FLAG: --feature-gates=""
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487095    5415 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487100    5415 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487105    5415 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487110    5415 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487116    5415 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487121    5415 flags.go:64] FLAG: --help="false"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487125    5415 flags.go:64] FLAG: --hostname-override="ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487131    5415 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487136    5415 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487141    5415 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487147    5415 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487153    5415 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487158    5415 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 03:07:38.494259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487163    5415 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487169    5415 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487176    5415 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487180    5415 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487185    5415 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487190    5415 flags.go:64] FLAG: --kube-reserved=""
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487195    5415 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487200    5415 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487204    5415 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487209    5415 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487213    5415 flags.go:64] FLAG: --lock-file=""
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487218    5415 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487223    5415 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487228    5415 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487243    5415 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487247    5415 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487252    5415 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487257    5415 flags.go:64] FLAG: --logging-format="text"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487261    5415 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487267    5415 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487272    5415 flags.go:64] FLAG: --manifest-url=""
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487277    5415 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487284    5415 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487289    5415 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487296    5415 flags.go:64] FLAG: --max-pods="110"
Oct 31 03:07:38.494997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487301    5415 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487306    5415 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487311    5415 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487316    5415 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487320    5415 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487325    5415 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487330    5415 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487341    5415 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487347    5415 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487354    5415 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487359    5415 flags.go:64] FLAG: --pod-cidr=""
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487365    5415 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487370    5415 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487375    5415 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487380    5415 flags.go:64] FLAG: --port="10250"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487385    5415 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487390    5415 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0a978aae77f6894a2"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487395    5415 flags.go:64] FLAG: --qos-reserved=""
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487400    5415 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487405    5415 flags.go:64] FLAG: --register-node="true"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487410    5415 flags.go:64] FLAG: --register-with-taints=""
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487416    5415 flags.go:64] FLAG: --registry-burst="10"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487421    5415 flags.go:64] FLAG: --registry-qps="5"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487425    5415 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487430    5415 flags.go:64] FLAG: --reserved-memory=""
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487436    5415 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 03:07:38.495707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487441    5415 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487446    5415 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487450    5415 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487455    5415 flags.go:64] FLAG: --runonce="false"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487460    5415 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487464    5415 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487470    5415 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487475    5415 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487479    5415 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487484    5415 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487489    5415 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487494    5415 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487499    5415 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487503    5415 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487509    5415 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487515    5415 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487520    5415 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487525    5415 flags.go:64] FLAG: --system-cgroups=""
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487530    5415 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487539    5415 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487544    5415 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487549    5415 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487562    5415 flags.go:64] FLAG: --tls-min-version=""
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487567    5415 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487572    5415 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 03:07:38.496445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487577    5415 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487581    5415 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487587    5415 flags.go:64] FLAG: --v="2"
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487594    5415 flags.go:64] FLAG: --version="false"
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487600    5415 flags.go:64] FLAG: --vmodule=""
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487607    5415 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.487612    5415 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487824    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487830    5415 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487834    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487839    5415 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487843    5415 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487847    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487851    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487856    5415 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487861    5415 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487865    5415 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487869    5415 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487887    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487892    5415 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487896    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:38.497135 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487902    5415 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487908    5415 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487912    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487916    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487920    5415 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487924    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487928    5415 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487933    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487937    5415 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487941    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487945    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487950    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487954    5415 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487959    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487963    5415 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487967    5415 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487971    5415 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487975    5415 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487979    5415 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:38.497784 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487984    5415 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487988    5415 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487992    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.487996    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488000    5415 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488004    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488008    5415 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488014    5415 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488018    5415 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488023    5415 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488027    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488031    5415 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488035    5415 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488043    5415 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488048    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488052    5415 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488056    5415 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488060    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488065    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:38.498376 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488071    5415 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488076    5415 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488081    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488085    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488089    5415 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488094    5415 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488098    5415 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488102    5415 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488106    5415 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488110    5415 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488114    5415 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488118    5415 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488124    5415 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488130    5415 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488134    5415 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488139    5415 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488144    5415 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488149    5415 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488154    5415 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488158    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:38.498997 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488162    5415 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488167    5415 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488172    5415 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488176    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488180    5415 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488184    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488190    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488194    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488198    5415 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488202    5415 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488206    5415 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488210    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488214    5415 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488218    5415 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488223    5415 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488228    5415 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488232    5415 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488236    5415 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488241    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:38.499592 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488245    5415 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488249    5415 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488253    5415 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488257    5415 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488261    5415 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488265    5415 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488269    5415 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488273    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488277    5415 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488281    5415 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488286    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488290    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488294    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488298    5415 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488302    5415 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.488344    5415 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:07:38.500185 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488399    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488406    5415 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488410    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488415    5415 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488421    5415 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488425    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488430    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488434    5415 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488438    5415 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488443    5415 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488447    5415 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488450    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488454    5415 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488459    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488463    5415 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488467    5415 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488471    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488475    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488479    5415 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488483    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:38.500731 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488488    5415 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488492    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488496    5415 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488500    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488504    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488508    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488512    5415 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488516    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488521    5415 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488525    5415 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488529    5415 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488535    5415 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488541    5415 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488545    5415 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488549    5415 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488553    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488558    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488563    5415 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488567    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:38.501363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488571    5415 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488575    5415 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488579    5415 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488583    5415 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488588    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488592    5415 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488596    5415 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488600    5415 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488604    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488608    5415 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488612    5415 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488617    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488622    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488629    5415 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488635    5415 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488640    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488644    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488648    5415 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488653    5415 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:38.501895 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488657    5415 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488661    5415 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488665    5415 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488669    5415 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488673    5415 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488679    5415 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488686    5415 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488691    5415 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488696    5415 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488700    5415 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488706    5415 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488710    5415 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488714    5415 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488718    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488722    5415 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488727    5415 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488731    5415 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488735    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488740    5415 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:38.502426 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488744    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488748    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488752    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488757    5415 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488761    5415 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488765    5415 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488769    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488773    5415 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488777    5415 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488781    5415 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488785    5415 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488789    5415 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488793    5415 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488798    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488802    5415 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488806    5415 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488810    5415 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488814    5415 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488820    5415 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488825    5415 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:38.503119 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488829    5415 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488834    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488838    5415 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488843    5415 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488848    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488852    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488856    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488862    5415 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.488867    5415 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.494506    5415 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.494528    5415 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.494549    5415 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.494554    5415 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494638    5415 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494646    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494651    5415 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494656    5415 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494660    5415 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494665    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494669    5415 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:38.503749 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494672    5415 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494675    5415 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494680    5415 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494684    5415 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494689    5415 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494693    5415 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494696    5415 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494699    5415 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494702    5415 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494704    5415 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494707    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494710    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494713    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494715    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494718    5415 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494721    5415 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494724    5415 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494727    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494729    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494733    5415 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:38.504413 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494737    5415 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494740    5415 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494745    5415 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494749    5415 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494754    5415 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494759    5415 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494764    5415 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494768    5415 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494773    5415 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494777    5415 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494782    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494786    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494791    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494795    5415 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494799    5415 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494803    5415 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494808    5415 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494812    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494815    5415 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:38.505001 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494818    5415 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494821    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494824    5415 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494827    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494830    5415 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494833    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494835    5415 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494838    5415 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494840    5415 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494843    5415 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494848    5415 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494852    5415 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494855    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494858    5415 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494861    5415 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494864    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494867    5415 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494869    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494899    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:38.505567 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494905    5415 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494909    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494911    5415 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494914    5415 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494919    5415 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494923    5415 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494925    5415 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494928    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494931    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494934    5415 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494936    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494939    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494942    5415 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494944    5415 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494948    5415 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494951    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494953    5415 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494956    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494959    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494962    5415 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:38.506146 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494966    5415 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494970    5415 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494974    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494978    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494983    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494987    5415 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494991    5415 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494995    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.494998    5415 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495001    5415 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495004    5415 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495007    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495010    5415 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495013    5415 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495015    5415 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495018    5415 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495021    5415 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495024    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495026    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495029    5415 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:38.506751 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495032    5415 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.495065    5415 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495235    5415 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495243    5415 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495246    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495249    5415 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495252    5415 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495255    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495258    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495261    5415 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495263    5415 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495266    5415 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495269    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495272    5415 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495274    5415 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495277    5415 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:38.507440 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495280    5415 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495282    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495287    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495291    5415 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495296    5415 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495300    5415 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495302    5415 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495305    5415 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495308    5415 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495310    5415 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495313    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495316    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495319    5415 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495321    5415 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495324    5415 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495327    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495329    5415 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495332    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495334    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:38.507912 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495338    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495340    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495343    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495345    5415 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495348    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495352    5415 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495355    5415 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495358    5415 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495360    5415 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495363    5415 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495366    5415 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495368    5415 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495371    5415 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495374    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495376    5415 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495379    5415 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495382    5415 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495385    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495388    5415 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495390    5415 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:38.508401 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495393    5415 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495396    5415 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495398    5415 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495401    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495405    5415 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495409    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495413    5415 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495419    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495423    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495428    5415 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495433    5415 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495438    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495442    5415 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495447    5415 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495451    5415 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495456    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495461    5415 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495464    5415 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495467    5415 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495470    5415 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:38.509010 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495473    5415 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495476    5415 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495479    5415 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495481    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495485    5415 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495489    5415 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495492    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495495    5415 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495499    5415 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495502    5415 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495505    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495507    5415 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495509    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495512    5415 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495515    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495517    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495520    5415 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495523    5415 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495526    5415 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:38.509607 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495529    5415 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495532    5415 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495535    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495539    5415 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495543    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495546    5415 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495549    5415 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495552    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495554    5415 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495557    5415 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495559    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495562    5415 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495565    5415 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495568    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.495597    5415 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:07:38.510236 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495648    5415 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495653    5415 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495656    5415 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495659    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495664    5415 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495668    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495671    5415 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495674    5415 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495677    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495680    5415 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495682    5415 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495685    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495688    5415 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495691    5415 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495694    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495697    5415 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495699    5415 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495702    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495705    5415 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:38.510663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495707    5415 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495710    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495713    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495716    5415 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495719    5415 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495721    5415 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495724    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495727    5415 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495730    5415 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495732    5415 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495735    5415 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495738    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495741    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495743    5415 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495746    5415 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495749    5415 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495751    5415 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495754    5415 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495756    5415 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:38.511339 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495759    5415 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495761    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495764    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495769    5415 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495772    5415 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495775    5415 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495779    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495783    5415 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495788    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495792    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495796    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495800    5415 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495805    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495809    5415 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495813    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495817    5415 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495821    5415 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495825    5415 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495828    5415 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:38.512040 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495831    5415 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495834    5415 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495837    5415 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495839    5415 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495842    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495845    5415 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495848    5415 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495851    5415 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495854    5415 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495857    5415 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495859    5415 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495862    5415 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495865    5415 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495867    5415 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495870    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495887    5415 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495895    5415 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495898    5415 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495901    5415 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495904    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:38.512647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495906    5415 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495909    5415 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495913    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495916    5415 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495918    5415 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495921    5415 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495924    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495926    5415 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495929    5415 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495931    5415 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495934    5415 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495937    5415 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495940    5415 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495943    5415 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495946    5415 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495950    5415 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495953    5415 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495956    5415 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495960    5415 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:38.513282 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495963    5415 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495966    5415 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495969    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495971    5415 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495974    5415 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495977    5415 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495981    5415 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495987    5415 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495992    5415 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:38.495999    5415 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.496085    5415 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.497287    5415 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.497355    5415 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.497558    5415 server.go:1014] "Starting client certificate rotation"
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.497698    5415 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.497766    5415 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:01:24 +0000 UTC" deadline="2028-06-05 05:30:04.987983004 +0000 UTC"
Oct 31 03:07:38.513840 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.497811    5415 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13994h22m26.490175875s"
Oct 31 03:07:38.514387 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.498193    5415 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:07:38.514387 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.499543    5415 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:07:38.514387 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.500948    5415 log.go:25] "Validated CRI v1 runtime API"
Oct 31 03:07:38.514387 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.502926    5415 log.go:25] "Validated CRI v1 image API"
Oct 31 03:07:38.514387 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.503837    5415 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 03:07:38.514387 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.506922    5415 fs.go:135] Filesystem UUIDs: map[0a526239-b7b9-4296-8a4e-9c591927164b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fedca4f8-ac5a-4b7d-ad03-1fa557183428:/dev/nvme0n1p4]
Oct 31 03:07:38.514650 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.506952    5415 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ce09138491b87298dbe7e4fbcf3a24da34384e2c8b444ce6546e00ceb27c792/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ce09138491b87298dbe7e4fbcf3a24da34384e2c8b444ce6546e00ceb27c792/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0eb0e92f85052d88c64727b4db314b5a5875543c670e6adec9799e6448e3edf4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0eb0e92f85052d88c64727b4db314b5a5875543c670e6adec9799e6448e3edf4/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fbf76a6d280d81147161a5c521406c20a8b3f4feeb53eb9d38839dacf7e4040/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fbf76a6d280d81147161a5c521406c20a8b3f4feeb53eb9d38839dacf7e4040/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1088bb71c93be618fad92d4b06d39e972f2693be186ddc175d1f790569d4d0bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1088bb71c93be618fad92d4b06d39e972f2693be186ddc175d1f790569d4d0bb/userdata/shm major:0 minor:318 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1559f59a3675c2e3d2c6133bb4ebf1f36a6d8183bb017536f51686ab1fb3e1b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1559f59a3675c2e3d2c6133bb4ebf1f36a6d8183bb017536f51686ab1fb3e1b5/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/482d8994a27be3a92c38394df337d8d0450d366ebd3bd3a47535b14a4d4be0da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/482d8994a27be3a92c38394df337d8d0450d366ebd3bd3a47535b14a4d4be0da/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e053babb6df547e91d28ea281ca9956a3bea569d86749e30b2b49dca2c973f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e053babb6df547e91d28ea281ca9956a3bea569d86749e30b2b49dca2c973f3/userdata/shm major:0 minor:394 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a19334855370632f158cab70d5975524cf2ce6394dcc4c55839ea58a91013d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a19334855370632f158cab70d5975524cf2ce6394dcc4c55839ea58a91013d4/userdata/shm major:0 minor:345 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/622440e093a3efaedfc3d60aabc00f5694e0f8d0981adecfad62281f028e47c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/622440e093a3efaedfc3d60aabc00f5694e0f8d0981adecfad62281f028e47c3/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6584704ea7674d07cd35fed8ec485303303ce0e50a7fefa74cc717303f48f534/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6584704ea7674d07cd35fed8ec485303303ce0e50a7fefa74cc717303f48f534/userdata/shm major:0 minor:267 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b45609908eb3f178730eee8e8d374b1e4a030142d4c973e9613e574c2d5bd1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b45609908eb3f178730eee8e8d374b1e4a030142d4c973e9613e574c2d5bd1a/userdata/shm major:0 minor:262 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d1c1147925b96371116ba5e77a174d636ff3c84082a86c26a4f49ce8f9317e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d1c1147925b96371116ba5e77a174d636ff3c84082a86c26a4f49ce8f9317e2/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c52cad3c746ec9eff866d2b783fca4e2103ec9d11e7b1006c24f1873aec35d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c52cad3c746ec9eff866d2b783fca4e2103ec9d11e7b1006c24f1873aec35d5/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dda0ac8387681a083e0ccbaa2c30e7472baba88429468c1ce6d7bde63099e5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dda0ac8387681a083e0ccbaa2c30e7472baba88429468c1ce6d7bde63099e5a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94a2cdb3b61e63680197f9105bfda11dc1cc8e2dd657361fc4382ddaab4fcc67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94a2cdb3b61e63680197f9105bfda11dc1cc8e2dd657361fc4382ddaab4fcc67/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1dffbd195d3834921291b32d6ee069053f5d1df3bdad560e047c130664e132a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1dffbd195d3834921291b32d6ee069053f5d1df3bdad560e047c130664e132a/userdata/shm major:0 minor:388 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac801bb14592f0e5676ca3681991887116327cdca91a73bbd1ee6e38e91b8bb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac801bb14592f0e5676ca3681991887116327cdca91a73bbd1ee6e38e91b8bb2/userdata/shm major:0 minor:251 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/add7b8d0710d0f7e7e60b81bbc250a61606f4fc694358c7f67940db9debef870/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/add7b8d0710d0f7e7e60b81bbc250a61606f4fc694358c7f67940db9debef870/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b73281d3fe33cea4e1073bb7e61c6266744c827e96a7af1e468af3cbb416f2f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b73281d3fe33cea4e1073bb7e61c6266744c827e96a7af1e468af3cbb416f2f1/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca9219ff27b1e5924dbd2574c0c2526c859a7b2880c484e6092d27630adb2dd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca9219ff27b1e5924dbd2574c0c2526c859a7b2880c484e6092d27630adb2dd0/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0bc205ffae2f9b64253bd01c30a7391cf6e4fd6e62ed327046110cc77267bd3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0bc205ffae2f9b64253bd01c30a7391cf6e4fd6e62ed327046110cc77267bd3/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef787f9a5262b613175feccfeb02e9b6716696ed5ef06b5dd0299392ac5f5fff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef787f9a5262b613175feccfeb02e9b6716696ed5ef06b5dd0299392ac5f5fff/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2e709eb582e6a911923218b63ea491261856e717672fb11192fe81149fdd15c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2e709eb582e6a911923218b63ea491261856e717672fb11192fe81149fdd15c/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc964d0c9ad099f2e03313d5decc2d7cc6de365d8fff1fcc2a1771172aede414/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc964d0c9ad099f2e03313d5decc2d7cc6de365d8fff1fcc2a1771172aede414/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd7c99e40f4ebc498610ab29dfdb8dc6e44d42a8645542070173227f0ad8d6c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd7c99e40f4ebc498610ab29dfdb8dc6e44d42a8645542070173227f0ad8d6c0/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0077be45-ce2b-46f0-8fe4-e881254f6386/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/0077be45-ce2b-46f0-8fe4-e881254f6386/volumes/kubernetes.io~secret/tls-certificates major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0af7ee9f-95f0-4632-989a-2beaf0fba3e3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0af7ee9f-95f0-4632-989a-2beaf0fba3e3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0af7ee9f-95f0-4632-989a-2beaf0fba3e3/volumes/kubernetes.io~projected/kube-api-access-lclzk:{mountpoint:/var/lib/kubelet/pods/0af7ee9f-95f0-4632-989a-2beaf0fba3e3/volumes/kubernetes.io~projected/kube-api-access-lclzk major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0af7ee9f-95f0-4632-989a-2beaf0fba3e3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0af7ee9f-95f0-4632-989a-2beaf0fba3e3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fbec67c-68ae-48f9-b780-bff7931da269/volumes/kubernetes.io~projected/kube-api-access-4dnpf:{mountpoint:/var/lib/kubelet/pods/0fbec67c-68ae-48f9-b780-bff7931da269/volumes/kubernetes.io~projected/kube-api-access-4dnpf major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fbec67c-68ae-48f9-b780-bff7931da269/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/0fbec67c-68ae-48f9-b780-bff7931da269/volumes/kubernetes.io~secret/metrics-certs major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16799a99-285f-40d5-acbb-a8fbe0d7a2d9/volumes/kubernetes.io~projected/kube-api-access-22m44:{mountpoint:/var/lib/kubelet/pods/16799a99-285f-40d5-acbb-a8fbe0d7a2d9/volumes/kubernetes.io~projected/kube-api-access-22m44 major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16799a99-285f-40d5-acbb-a8fbe0d7a2d9/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/16799a99-285f-40d5-acbb-a8fbe0d7a2d9/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16799a99-285f-40d5-acbb-a8fbe0d7a2d9/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/16799a99-285f-40d5-acbb-a8fbe0d7a2d9/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/177793bf-7bb1-4b0b-adb6-ba40483ae237/volumes/kubernetes.io~projected/kube-api-access-b4q5r:{mountpoint:/var/lib/kubelet/pods/177793bf-7bb1-4b0b-adb6-ba40483ae237/volumes/kubernetes.io~projected/kube-api-access-b4q5r major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/177793bf-7bb1-4b0b-adb6-ba40483ae237/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/177793bf-7bb1-4b0b-adb6-ba40483ae237/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/177793bf-7bb1-4b0b-adb6-ba40483ae237/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/177793bf-7bb1-4b0b-adb6-ba40483ae237/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9/volumes/kubernetes.io~projected/kube-api-access-d78tf:{mountpoint:/var/lib/kubelet/pods/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9/volumes/kubernetes.io~projected/kube-api-access-d78tf major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2941755b-c963-4514-a42a-c918a8283273/volumes/kubernetes.io~projected/kube-api-access-9m5bp:{mountpoint:/var/lib/kubelet/pods/2941755b-c963-4514-a42a-c918a8283273/volumes/kubernetes.io~projected/kube-api-access-9m5bp major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2941755b-c963-4514-a42a-c918a8283273/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2941755b-c963-4514-a42a-c918a8283273/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2941755b-c963-4514-a42a-c918a8283273/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/2941755b-c963-4514-a42a-c918a8283273/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/382f0f50-a8e4-463d-a64c-710cfffdc963/volumes/kubernetes.io~projected/kube-api-access-tknn7:{mountpoint:/var/lib/kubelet/pods/382f0f50-a8e4-463d-a64c-710cfffdc963/volumes/kubernetes.io~projected/kube-api-access-tknn7 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cd79b48-c38e-4822-86b5-4b7c3c858fb6/volumes/kubernetes.io~projected/kube-api-access-rnds5:{mountpoint:/var/lib/kubelet/pods/4cd79b48-c38e-4822-86b5-4b7c3c858fb6/volumes/kubernetes.io~projected/kube-api-access-rnds5 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cb392b3-c5a6-4ae1-af06-82f167ed3c27/volumes/kubernetes.io~projected/kube-api-access-k5j7c:{mountpoint:/var/lib/kubelet/pods/6cb392b3-c5a6-4ae1-af06-82f167ed3c27/volumes/kubernetes.io~projected/kube-api-access-k5j7c major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/761f94e4-2681-4253-abb6-898d5f67aa98/volumes/kubernetes.io~projected/kube-api-access-wjbgf:{mountpoint:/var/lib/kubelet/pods/761f94e4-2681-4253-abb6-898d5f67aa98/volumes/kubernetes.io~projected/kube-api-access-wjbgf major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/761f94e4-2681-4253-abb6-898d5f67aa98/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/761f94e4-2681-4253-abb6-898d5f67aa98/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fdb73b3-0430-4d0c-a813-50a1beebdb12/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8fdb73b3-0430-4d0c-a813-50a1beebdb12/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fdb73b3-0430-4d0c-a813-50a1beebdb12/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8fdb73b3-0430-4d0c-a813-50a1beebdb12/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fdb73b3-0430-4d0c-a813-50a1beebdb12/volumes/kubernetes.io~projected/kube-api-access-sf4cc:{mountpoint:/var/lib/kubelet/pods/8fdb73b3-0430-4d0c-a813-50a1beebdb12/volumes/kubernetes.io~projected/kube-api-access-sf4cc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a18739ba-0914-4b52-bcda-4a5fafe22225/volumes/kubernetes.io~projected/kube-api-access-g8bd5:{mountpoint:/var/lib/kubelet/pods/a18739ba-0914-4b52-bcda-4a5fafe22225/volumes/kubernetes.io~projected/kube-api-access-g8bd5 major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a18739ba-0914-4b52-bcda-4a5fafe22225/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/a18739ba-0914-4b52-bcda-4a5fafe22225/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4d955b0-1418-499f-9036-961443f1e130/volumes/kubernetes.io~projected/kube-api-access-n965n:{mountpoint:/var/lib/kubelet/pods/a4d955b0-1418-499f-9036-961443f1e130/volumes/kubernetes.io~projected/kube-api-access-n965n major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4d955b0-1418-499f-9036-961443f1e130/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/a4d955b0-1418-499f-9036-961443f1e130/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4d955b0-1418-499f-9036-961443f1e130/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/a4d955b0-1418-499f-9036-961443f1e130/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4d955b0-1418-499f-9036-961443f1e130/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/a4d955b0-1418-499f-9036-961443f1e130/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2e85a91-0400-4117-a221-cc7961fd3ba0/volumes/kubernetes.io~projected/kube-api-access-tktxh:{mountpoint:/var/lib/kubelet/pods/b2e85a91-0400-4117-a221-cc7961fd3ba0/volumes/kubernetes.io~projected/kube-api-access-tktxh major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684/volumes/kubernetes.io~projected/kube-api-access-6zvxt:{mountpoint:/var/lib/kubelet/pods/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684/volumes/kubernetes.io~projected/kube-api-access-6zvxt major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c88db770-eb3f-4879-8a41-7e4d7d87fe0e/volumes/kubernetes.io~projected/kube-api-access-n2fzj:{mountpoint:/var/lib/kubelet/pods/c88db770-eb3f-4879-8a41-7e4d7d87fe0e/volumes/kubernetes.io~projected/kube-api-access-n2fzj major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c88db770-eb3f-4879-8a41-7e4d7d87fe0e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/c88db770-eb3f-4879-8a41-7e4d7d87fe0e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d35c8150-ce21-4660-ad5d-e5cc1fce684b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d35c8150-ce21-4660-ad5d-e5cc1fce684b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da297bce-ca72-4dea-b02c-bcda6f401ebe/volumes/kubernetes.io~projected/kube-api-access-xvlpv:{mountpoint:/var/lib/kubelet/pods/da297bce-ca72-4dea-b02c-bcda6f401ebe/volumes/kubernetes.io~projected/kube-api-access-xvlpv major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da297bce-ca72-4dea-b02c-bcda6f401ebe/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/da297bce-ca72-4dea-b02c-bcda6f401ebe/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da297bce-ca72-4dea-b02c-bcda6f401ebe/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/da297bce-ca72-4dea-b02c-bcda6f401ebe/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e387f3b0-3991-4570-895b-8b59c01c8f36/volumes/kubernetes.io~projected/kube-api-access-9695l:{mountpoint:/var/lib/kubelet/pods/e387f3b0-3991-4570-895b-8b59c01c8f36/volumes/kubernetes.io~projected/kube-api-access-9695l major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e387f3b0-3991-4570-895b-8b59c01c8f36/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e387f3b0-3991-4570-895b-8b59c01c8f36/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a/volumes/kubernetes.io~projected/kube-api-access-nkbch:{mountpoint:/var/lib/kubelet/pods/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a/volumes/kubernetes.io~projected/kube-api-access-nkbch major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea79f848-e58a-463b-8c7f-d6f6dd0125b3/volumes/kubernetes.io~projected/kube-api-access-94qnj:{mountpoint:/var/lib/kubelet/pods/ea79f848-e58a-463b-8c7f-d6f6dd0125b3/volumes/kubernetes.io~projected/kube-api-access-94qnj major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad83c1a-3d12-44d3-9164-4c015d044f3e/volumes/kubernetes.io~projected/kube-api-access-r8c5k:{mountpoint:/var/lib/kubelet/pods/fad83c1a-3d12-44d3-9164-4c015d044f3e/volumes/kubernetes.io~projected/kube-api-access-r8c5k major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/70e97985028ac4c75a1d3e2a931597538b6832ce42b0e4f0bb96b9925245f3ae/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1b669e377f95b654e8682d2f0914de65480786ea8556dcebcb6699ccdf1409ba/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f5fe1cc90372fef31df6eb9376efff783ddef0f63964dd90e5b1a939f4e49ce7/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/677cc9437df45641f143d46cb6b1d1c8bca6f8174830a5537ce3508c6395db34/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/33760ef7bdabe023f29fbd843c3432b29cb259447c27abb78b799cc15820cd5d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c0e1667fa7014d49f25b316e0d5f8f97d80a177be8bbb0f40b0614bebe4c165c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/ec2fff6652dafd28904a2679a7275163085e208a06c114becb483846027575c8/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5ca96b56f7fb1778f4d4d86dd52e9364cfdc0741ef2f7210444d81662b2677e6/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/63eb27f22271e68e60438c16749fe68684619e41021fd68bcbfa0944dc2a8d0c/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/c597326a7dfcf0464c49af40d0a7367172f51a009a6f8abb3652492009a28749/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/ca201a069d237f7067c5cdf7d016cb5ec156259fdb398f99c7de344fc9b9231b/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/e50dc58941181654a0c87ac7f0620c1a99e97a715d48be4cae2a6ab8ad0c2449/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a385c3becd3ba6c50e5dedf47aa08aafc87417c04cf270e843105e2bbab0142c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/dc0a171bf848e841f3096c608fc6da434aa192c32dc33134a1fde6b594909dbb/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/248a8f82ccee10f20fd630d512e25bffbf3721fa8b907c1bac847a31de489b49/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8867cc56a65bed7fc97de5647ed30698e3c58a1705d66c54d70dacd72db8cd2a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c7717acaf9febabc9986a4974afd6c50d9cd56a39740c5f46a06fb517d3e41c9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/8881149bbdea3962eb24861b12911e7ec05e34cb798756f1b1e0bee8be483373/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/eb3be35ba6dc4be4fda593b3b7defa4a5d48ee9169fb54125e5c4a6600c83aca/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/65178f3a2c93a712dab6724f036a9e48e460b48b84f45b2748f82e25fe150d53/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/fb0f73d29acbe1a609a319152decd9d48cc3179dcb7c24fa1e855cd7c444fc3d/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c89f5374f89227863618be5d1d001555b103a24f2b1da4d75aff6ad670947117/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/9511806f4301f7391ce28f9380d1bd9790fe86509d2d4c623b40c903a843b162/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/e79f13345820d9a5de896e571fe856b6e4e09141cbb40f0c0a05de5d1e6829d9/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/627d13072f44ca82fa2ca9eb16aabbdbf4122d543434433f06ba3c0de5930bf9/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/871ca874466e0d9a1454568d562239a2d0ba228006979df050983bc247fdc62b/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/535fa11e0753784a6af091bd7195ddd0a0dabfdc0006d9d2d6b3f674b62b3e6c/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/3b41f24cc1d208f6819afc2f86ed15eddd246bdc6b5b5b492833b47a8a936bc7/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/275491d714ec0f316e1792743ffc91c9607d849ee48bcb291403f41b1060c4d6/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/3f5cae9cc4571b022e44b71aa5b07d085ad3fc72159c188422823182f2f45923/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/d1426ab7360f4c85173706f629b157e2a61bed453e203eb12c2c0ccff183f223/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/c494a6579d71ca85abe6ffabf6352c726a5626ec24410d20644a0e84806b54eb/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/9201f936660f3ef70f16d498d1de107ca9ee146bde0b6f5f0220b36298a97155/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/9030f56e0f1f864116865297190f95f5e6e2c68706aa881d6f737594a0812306/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/21673bde3a01facf10dde6e591f3d8c334518c2d4c477851cb515465f333e411/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/fe0169a6ba817fb8daf6b737644b9baf2bd134b6c4a0ea2b88ffd9c8f03bb55c/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/853997ac45744c5d149dcda31a80412486bee795137799848b1bfa7ff5397038/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/021918c4cd883fdcad43eb993c9dcdfe69389c8911e7610f015b0fba752966d4/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/75ec210dd4c59e08fdec87b99d4b8036559302069128882028d61ea7a4fc4437/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/6442d932166615cda2e4c0508486a2d97bfd472ad418173cefb4d6d620456e67/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/83122c679791ff81c33df67646c78f0c0f42ade80a4997ce6c7abe2634242eb4/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/a28dec2bac1132b25428ff0ab253a73b931eed524ded2a19337ec0cbd8fc60cb/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/ab216c4d9ef4624dab2e9a8c0a9586447f73ca939643ec2ea7d6194d02e5fd1c/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/851f1b3b69e74b8f8ce6dd380da00606165f1c1693d4067f833abc112d0dc225/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/1a92b6daaa83901f03855cd98ddbd569ee74a9f328d2abf99903dfa217285a04/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/71012516801d774db6f95231301f90404e6544cd7e98a7ba8803edd46f249f06/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/dd4b4d17a50418e83f84592ba720421e94ac9d4d829f60ee9e048ce9748e88d9/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/069a3fedecad8d32bba08a739b368faa14d335713e1c09f4dadcc7135c16c1e3/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/c03f5326654e88fffc9688cae55bd0ab394ba04576453ab0e2e7bf3d037e3a0c/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/f8a6368667b629ea4a64136ea7bd1391e9730c0f3a1e0f97aea597f4c329532a/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/aab863eb7300646ca3cb07e259f864a494530361cd1fb45ba11ca174982f2062/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/2bdc07a8a6c95c1eb668a8b7203fefdb84dc3fc7167a8881c69a64ea40e50955/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/524d0bc60887ebda809ab00d72ea5c120438d1f26aed166956545edc8de3c895/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/ce401a708a0f7e81de4a6c28a51df9ce9173139e6ef6df3f8c845170564d51a8/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/bd49014aff782f790001fa114387c4f15e48ffce1953e8de4e9197c1846414d8/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/76a3e2cf9cc788fcbb05ee4e3087712c15148544549230d201f8c414b29ba95b/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/1e5bc8eccf0addffaa407d953a4042c3d684cd55d567665e576041baeb6744b5/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/ee83ec5ce320582e7f0904372821b0239d22c4914825fb9c2082ac3bd6f0f07f/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/25d5a54bb372e2caa40d14c391bb0dff9b5a164b0dab5aa24e9db97faf64bb84/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/27fc385af52cecc36a82e9bf4f4485f6ce6a158fef52cd1f3724a0a5a21dff97/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/adcaf893f3f012ccf9be410958eefe1fdfd1a2b46492a47fad0b061544952eb0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/5d4160786a487f1f4f47fbf9b35b74de392d65fbe0bbd7c241a89e7d5854ed1a/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/2c72b62b658a3b5a08988c2b36cae946982aba226514aa95d706185a32a667fe/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/ab9cdf9d17bfc1ba3acf973a9d3a534d5c9c5440649500e205167e3a731caf93/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/d96dfa8378d84ccaf67b8bb0cf64e5b46aa61dc4c388cb7e954d3af84079e07d/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/b020ae560a4cb8b2050885d12c70c3ac6f14f2acfcc76e1b623e0faef4acd662/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/b395a625fa47b9720377771fa3ee0d21728632fab833dbf78234f0fe0b00d201/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 31 03:07:38.521238 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.520374    5415 manager.go:217] Machine: {Timestamp:2026-10-31 03:07:38.51937902 +0000 UTC m=+0.079858743 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099797 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c17d45e05717897c12e1aed72c632 SystemUUID:ec2c17d4-5e05-7178-97c1-2e1aed72c632 BootID:4b3ec0f4-ec15-41f5-baba-760f45e2bfa3 Filesystems:[{Device:/var/lib/kubelet/pods/2941755b-c963-4514-a42a-c918a8283273/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/da297bce-ca72-4dea-b02c-bcda6f401ebe/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/177793bf-7bb1-4b0b-adb6-ba40483ae237/volumes/kubernetes.io~projected/kube-api-access-b4q5r DeviceMajor:0 DeviceMinor:307 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fbec67c-68ae-48f9-b780-bff7931da269/volumes/kubernetes.io~projected/kube-api-access-4dnpf DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9/volumes/kubernetes.io~projected/kube-api-access-d78tf DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4d955b0-1418-499f-9036-961443f1e130/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:391 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e387f3b0-3991-4570-895b-8b59c01c8f36/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/177793bf-7bb1-4b0b-adb6-ba40483ae237/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:305 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1559f59a3675c2e3d2c6133bb4ebf1f36a6d8183bb017536f51686ab1fb3e1b5/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fbf76a6d280d81147161a5c521406c20a8b3f4feeb53eb9d38839dacf7e4040/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a/volumes/kubernetes.io~projected/kube-api-access-nkbch DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2e709eb582e6a911923218b63ea491261856e717672fb11192fe81149fdd15c/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b73281d3fe33cea4e1073bb7e61c6266744c827e96a7af1e468af3cbb416f2f1/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c88db770-eb3f-4879-8a41-7e4d7d87fe0e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0077be45-ce2b-46f0-8fe4-e881254f6386/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:247 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da297bce-ca72-4dea-b02c-bcda6f401ebe/volumes/kubernetes.io~projected/kube-api-access-xvlpv DeviceMajor:0 DeviceMinor:306 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fad83c1a-3d12-44d3-9164-4c015d044f3e/volumes/kubernetes.io~projected/kube-api-access-r8c5k DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e387f3b0-3991-4570-895b-8b59c01c8f36/volumes/kubernetes.io~projected/kube-api-access-9695l DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0eb0e92f85052d88c64727b4db314b5a5875543c670e6adec9799e6448e3edf4/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/622440e093a3efaedfc3d60aabc00f5694e0f8d0981adecfad62281f028e47c3/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16799a99-285f-40d5-acbb-a8fbe0d7a2d9/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:263 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/da297bce-ca72-4dea-b02c-bcda6f401ebe/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2941755b-c963-4514-a42a-c918a8283273/volumes/kubernetes.io~projected/kube-api-access-9m5bp DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/177793bf-7bb1-4b0b-adb6-ba40483ae237/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e053babb6df547e91d28ea281ca9956a3bea569d86749e30b2b49dca2c973f3/userdata/shm DeviceMajor:0 DeviceMinor:394 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/8fdb73b3-0430-4d0c-a813-50a1beebdb12/volumes/kubernetes.io~projected/kube-api-access-sf4cc DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0af7ee9f-95f0-4632-989a-2beaf0fba3e3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0af7ee9f-95f0-4632-989a-2beaf0fba3e3/volumes/kubernetes.io~projected/kube-api-access-lclzk DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16799a99-285f-40d5-acbb-a8fbe0d7a2d9/volumes/kubernetes.io~projected/kube-api-access-22m44 DeviceMajor:0 DeviceMinor:266 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/761f94e4-2681-4253-abb6-898d5f67aa98/volumes/kubernetes.io~projected/kube-api-access-wjbgf DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1dffbd195d3834921291b32d6ee069053f5d1df3bdad560e047c130664e132a/userdata/shm DeviceMajor:0 DeviceMinor:388 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/8fdb73b3-0430-4d0c-a813-50a1beebdb12/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b45609908eb3f178730eee8e8d374b1e4a030142d4c973e9613e574c2d5bd1a/userdata/shm DeviceMajor:0 DeviceMinor:262 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d35c8150-ce21-4660-ad5d-e5cc1fce684b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d1c1147925b96371116ba5e77a174d636ff3c84082a86c26a4f49ce8f9317e2/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8fdb73b3-0430-4d0c-a813-50a1beebdb12/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc964d0c9ad099f2e03313d5decc2d7cc6de365d8fff1fcc2a1771172aede414/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd7c99e40f4ebc498610ab29dfdb8dc6e44d42a8645542070173227f0ad8d6c0/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684/volumes/kubernetes.io~projected/kube-api-access-6zvxt DeviceMajor:0 DeviceMinor:344 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4d955b0-1418-499f-9036-961443f1e130/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:390 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2e85a91-0400-4117-a221-cc7961fd3ba0/volumes/kubernetes.io~projected/kube-api-access-tktxh DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6cb392b3-c5a6-4ae1-af06-82f167ed3c27/volumes/kubernetes.io~projected/kube-api-access-k5j7c DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2941755b-c963-4514-a42a-c918a8283273/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a18739ba-0914-4b52-bcda-4a5fafe22225/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:382 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a18739ba-0914-4b52-bcda-4a5fafe22225/volumes/kubernetes.io~projected/kube-api-access-g8bd5 DeviceMajor:0 DeviceMinor:387 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0bc205ffae2f9b64253bd01c30a7391cf6e4fd6e62ed327046110cc77267bd3/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16799a99-285f-40d5-acbb-a8fbe0d7a2d9/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:264 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6584704ea7674d07cd35fed8ec485303303ce0e50a7fefa74cc717303f48f534/userdata/shm DeviceMajor:0 DeviceMinor:267 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/482d8994a27be3a92c38394df337d8d0450d366ebd3bd3a47535b14a4d4be0da/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/382f0f50-a8e4-463d-a64c-710cfffdc963/volumes/kubernetes.io~projected/kube-api-access-tknn7 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/761f94e4-2681-4253-abb6-898d5f67aa98/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fbec67c-68ae-48f9-b780-bff7931da269/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:281 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ce09138491b87298dbe7e4fbcf3a24da34384e2c8b444ce6546e00ceb27c792/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0af7ee9f-95f0-4632-989a-2beaf0fba3e3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef787f9a5262b613175feccfeb02e9b6716696ed5ef06b5dd0299392ac5f5fff/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1088bb71c93be618fad92d4b06d39e972f2693be186ddc175d1f790569d4d0bb/userdata/shm DeviceMajor:0 DeviceMinor:318 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea79f848-e58a-463b-8c7f-d6f6dd0125b3/volumes/kubernetes.io~projected/kube-api-access-94qnj DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a19334855370632f158cab70d5975524cf2ce6394dcc4c55839ea58a91013d4/userdata/shm DeviceMajor:0 DeviceMinor:345 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4d955b0-1418-499f-9036-961443f1e130/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:392 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac801bb14592f0e5676ca3681991887116327cdca91a73bbd1ee6e38e91b8bb2/userdata/shm DeviceMajor:0 DeviceMinor:251 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4d955b0-1418-499f-9036-961443f1e130/volumes/kubernetes.io~projected/kube-api-access-n965n DeviceMajor:0 DeviceMinor:393 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4cd79b48-c38e-4822-86b5-4b7c3c858fb6/volumes/kubernetes.io~projected/kube-api-access-rnds5 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/add7b8d0710d0f7e7e60b81bbc250a61606f4fc694358c7f67940db9debef870/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c52cad3c746ec9eff866d2b783fca4e2103ec9d11e7b1006c24f1873aec35d5/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca9219ff27b1e5924dbd2574c0c2526c859a7b2880c484e6092d27630adb2dd0/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dda0ac8387681a083e0ccbaa2c30e7472baba88429468c1ce6d7bde63099e5a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94a2cdb3b61e63680197f9105bfda11dc1cc8e2dd657361fc4382ddaab4fcc67/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c88db770-eb3f-4879-8a41-7e4d7d87fe0e/volumes/kubernetes.io~projected/kube-api-access-n2fzj DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0ce09138491b872 MacAddress:d2:b8:71:c4:64:87 Speed:10000 Mtu:8901} {Name:0eb0e92f85052d8 MacAddress:aa:e1:aa:bf:4d:43 Speed:10000 Mtu:8901} {Name:0fbf76a6d280d81 MacAddress:62:b7:98:bc:29:c4 Speed:10000 Mtu:8901} {Name:1088bb71c93be61 MacAddress:1e:b2:75:07:9f:71 Speed:10000 Mtu:8901} {Name:4e053babb6df547 MacAddress:aa:ce:a0:6e:b8:a5 Speed:10000 Mtu:8901} {Name:5a1933485537063 MacAddress:2a:0b:28:98:3a:ef Speed:10000 Mtu:8901} {Name:6584704ea7674d0 MacAddress:86:44:5a:55:ae:f4 Speed:10000 Mtu:8901} {Name:7b45609908eb3f1 MacAddress:6a:0f:42:9b:54:d0 Speed:10000 Mtu:8901} {Name:7d1c1147925b963 MacAddress:42:fd:a5:f5:7a:9b Speed:10000 Mtu:8901} {Name:a1dffbd195d3834 MacAddress:46:84:ae:48:d3:93 Speed:10000 Mtu:8901} {Name:ac801bb14592f0e MacAddress:2e:5a:73:ba:70:5f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:cf:9d:b4:6d Speed:0 Mtu:9001} {Name:br-int MacAddress:0a:94:d0:6f:a0:fe Speed:0 Mtu:8901} {Name:d0bc205ffae2f9b MacAddress:72:71:c6:ef:37:31 Speed:10000 Mtu:8901} {Name:ef787f9a5262b61 MacAddress:0e:0b:6c:dc:1f:a2 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:cf:9d:b4:6d Speed:0 Mtu:9001} {Name:fc964d0c9ad099f MacAddress:fa:93:d5:f6:48:e8 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:4e:c8:a7:e3:ac:98 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:36:e2:6b:9d:37:e0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 03:07:38.521238 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.521241    5415 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 03:07:38.521782 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.521318    5415 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 03:07:38.522782 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.522739    5415 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 03:07:38.523039 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.522783    5415 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-164.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 03:07:38.523109 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.523045    5415 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 03:07:38.523109 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.523058    5415 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 03:07:38.523109 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.523078    5415 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:07:38.523109 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.523100    5415 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:07:38.523282 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.523271    5415 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 03:07:38.523532 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.523519    5415 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 03:07:38.523679 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.523668    5415 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 03:07:38.523725 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.523719    5415 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 03:07:38.523808 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.523733    5415 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 03:07:38.523808 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.523749    5415 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 03:07:38.523808 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.523763    5415 kubelet.go:405] "Adding apiserver pod source"
Oct 31 03:07:38.523808 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.523775    5415 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 03:07:38.524702 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.524685    5415 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 03:07:38.524702 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.524703    5415 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:07:38.525339 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.525319    5415 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 03:07:38.538368 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.538343    5415 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 03:07:38.538368 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.538363    5415 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:07:38.538585 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.538571    5415 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 03:07:38.538829 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.538813    5415 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 03:07:38.538919 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.538853    5415 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 03:07:38.539031 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539010    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 03:07:38.539031 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539033    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 03:07:38.539203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539041    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 03:07:38.539203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539047    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 03:07:38.539203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539054    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 03:07:38.539203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539060    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 03:07:38.539203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539068    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 03:07:38.539203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539078    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 03:07:38.539203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539089    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 03:07:38.539203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539096    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 03:07:38.539203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539107    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 03:07:38.539203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539118    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 03:07:38.539203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539152    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 03:07:38.539203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.539159    5415 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 03:07:38.543185 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.543165    5415 server.go:1263] "Started kubelet"
Oct 31 03:07:38.543341 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.543259    5415 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 03:07:38.543426 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.543379    5415 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 03:07:38.543468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.543450    5415 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 03:07:38.546058 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.543970    5415 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 03:07:38.546058 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.544633    5415 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 03:07:38.544227 ip-10-0-139-164 systemd[1]: Started Kubernetes Kubelet.
Oct 31 03:07:38.550556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.550520    5415 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 03:07:38.551212 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.551187    5415 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 03:07:38.551449 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.551427    5415 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 03:07:38.551529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.551452    5415 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 03:07:38.552946 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:07:38.551431    5415 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:07:38.552946 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.551509    5415 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 03:07:38.552946 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.551999    5415 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:01:25 +0000 UTC" deadline="2028-06-14 04:17:49.856427243 +0000 UTC"
Oct 31 03:07:38.552946 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.552038    5415 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14209h10m11.304392805s"
Oct 31 03:07:38.552946 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.552753    5415 factory.go:55] Registering systemd factory
Oct 31 03:07:38.552946 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.552777    5415 factory.go:223] Registration of the systemd container factory successfully
Oct 31 03:07:38.553273 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.553252    5415 factory.go:153] Registering CRI-O factory
Oct 31 03:07:38.553273 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.553266    5415 factory.go:223] Registration of the crio container factory successfully
Oct 31 03:07:38.553368 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.553328    5415 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 03:07:38.553368 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.553347    5415 factory.go:103] Registering Raw factory
Oct 31 03:07:38.553472 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.553371    5415 manager.go:1199] Started watching for new ooms in manager
Oct 31 03:07:38.554919 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.553704    5415 manager.go:319] Starting recovery of all containers
Oct 31 03:07:38.556743 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.556700    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbec67c-68ae-48f9-b780-bff7931da269" volumeName="kubernetes.io/projected/0fbec67c-68ae-48f9-b780-bff7931da269-kube-api-access-4dnpf" seLinuxMountContext=""
Oct 31 03:07:38.556896 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.556858    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="382f0f50-a8e4-463d-a64c-710cfffdc963" volumeName="kubernetes.io/configmap/382f0f50-a8e4-463d-a64c-710cfffdc963-cni-binary-copy" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.556907    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c88db770-eb3f-4879-8a41-7e4d7d87fe0e" volumeName="kubernetes.io/secret/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.556919    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da297bce-ca72-4dea-b02c-bcda6f401ebe" volumeName="kubernetes.io/projected/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-api-access-xvlpv" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.556928    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da297bce-ca72-4dea-b02c-bcda6f401ebe" volumeName="kubernetes.io/secret/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-tls" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.556936    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e387f3b0-3991-4570-895b-8b59c01c8f36" volumeName="kubernetes.io/secret/e387f3b0-3991-4570-895b-8b59c01c8f36-cert" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.556947    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbec67c-68ae-48f9-b780-bff7931da269" volumeName="kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.556959    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="177793bf-7bb1-4b0b-adb6-ba40483ae237" volumeName="kubernetes.io/configmap/177793bf-7bb1-4b0b-adb6-ba40483ae237-metrics-client-ca" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.556980    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="382f0f50-a8e4-463d-a64c-710cfffdc963" volumeName="kubernetes.io/projected/382f0f50-a8e4-463d-a64c-710cfffdc963-kube-api-access-tknn7" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.556989    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cb392b3-c5a6-4ae1-af06-82f167ed3c27" volumeName="kubernetes.io/projected/6cb392b3-c5a6-4ae1-af06-82f167ed3c27-kube-api-access-k5j7c" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.556996    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" volumeName="kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-oauth-serving-cert" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557006    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c88db770-eb3f-4879-8a41-7e4d7d87fe0e" volumeName="kubernetes.io/empty-dir/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-data-volume" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557019    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9c7f8e8-6c5e-4f69-bede-8e89d342d02a" volumeName="kubernetes.io/empty-dir/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-tmp-dir" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557033    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" volumeName="kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-config" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557051    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16799a99-285f-40d5-acbb-a8fbe0d7a2d9" volumeName="kubernetes.io/configmap/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-metrics-client-ca" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557060    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="177793bf-7bb1-4b0b-adb6-ba40483ae237" volumeName="kubernetes.io/configmap/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 31 03:07:38.557059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557068    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="177793bf-7bb1-4b0b-adb6-ba40483ae237" volumeName="kubernetes.io/projected/177793bf-7bb1-4b0b-adb6-ba40483ae237-kube-api-access-b4q5r" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557079    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2941755b-c963-4514-a42a-c918a8283273" volumeName="kubernetes.io/secret/2941755b-c963-4514-a42a-c918a8283273-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557090    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a18739ba-0914-4b52-bcda-4a5fafe22225" volumeName="kubernetes.io/projected/a18739ba-0914-4b52-bcda-4a5fafe22225-kube-api-access-g8bd5" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557101    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da297bce-ca72-4dea-b02c-bcda6f401ebe" volumeName="kubernetes.io/configmap/da297bce-ca72-4dea-b02c-bcda6f401ebe-metrics-client-ca" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557114    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0af7ee9f-95f0-4632-989a-2beaf0fba3e3" volumeName="kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557130    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0af7ee9f-95f0-4632-989a-2beaf0fba3e3" volumeName="kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-env-overrides" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557143    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16799a99-285f-40d5-acbb-a8fbe0d7a2d9" volumeName="kubernetes.io/secret/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557151    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17bd527b-9dd9-4fc6-86b3-57b5456c5a6b" volumeName="kubernetes.io/secret/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b-agent-certs" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557159    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad83c1a-3d12-44d3-9164-4c015d044f3e" volumeName="kubernetes.io/projected/fad83c1a-3d12-44d3-9164-4c015d044f3e-kube-api-access-r8c5k" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557167    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="177793bf-7bb1-4b0b-adb6-ba40483ae237" volumeName="kubernetes.io/empty-dir/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-textfile" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557177    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="761f94e4-2681-4253-abb6-898d5f67aa98" volumeName="kubernetes.io/empty-dir/761f94e4-2681-4253-abb6-898d5f67aa98-tmp-dir" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557184    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fdb73b3-0430-4d0c-a813-50a1beebdb12" volumeName="kubernetes.io/empty-dir/8fdb73b3-0430-4d0c-a813-50a1beebdb12-tmp" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557200    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a18739ba-0914-4b52-bcda-4a5fafe22225" volumeName="kubernetes.io/secret/a18739ba-0914-4b52-bcda-4a5fafe22225-monitoring-plugin-cert" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557243    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4d955b0-1418-499f-9036-961443f1e130" volumeName="kubernetes.io/empty-dir/a4d955b0-1418-499f-9036-961443f1e130-audit-log" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557260    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" volumeName="kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-oauth-config" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557271    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c88db770-eb3f-4879-8a41-7e4d7d87fe0e" volumeName="kubernetes.io/configmap/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557289    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20dcc2be-01c6-4294-bce6-3b9a77f0d3d9" volumeName="kubernetes.io/projected/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-kube-api-access-d78tf" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557296    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2941755b-c963-4514-a42a-c918a8283273" volumeName="kubernetes.io/configmap/2941755b-c963-4514-a42a-c918a8283273-metrics-client-ca" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557304    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="382f0f50-a8e4-463d-a64c-710cfffdc963" volumeName="kubernetes.io/configmap/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-daemon-config" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557312    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cd79b48-c38e-4822-86b5-4b7c3c858fb6" volumeName="kubernetes.io/configmap/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-serviceca" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557319    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c88db770-eb3f-4879-8a41-7e4d7d87fe0e" volumeName="kubernetes.io/projected/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-kube-api-access-n2fzj" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557328    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4d955b0-1418-499f-9036-961443f1e130" volumeName="kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-secret-metrics-server-tls" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557341    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" volumeName="kubernetes.io/projected/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-kube-api-access-6zvxt" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557352    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2941755b-c963-4514-a42a-c918a8283273" volumeName="kubernetes.io/projected/2941755b-c963-4514-a42a-c918a8283273-kube-api-access-9m5bp" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557363    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fdb73b3-0430-4d0c-a813-50a1beebdb12" volumeName="kubernetes.io/empty-dir/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-tuned" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557374    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2e85a91-0400-4117-a221-cc7961fd3ba0" volumeName="kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-cni-binary-copy" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557399    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="761f94e4-2681-4253-abb6-898d5f67aa98" volumeName="kubernetes.io/configmap/761f94e4-2681-4253-abb6-898d5f67aa98-config-volume" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557411    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4d955b0-1418-499f-9036-961443f1e130" volumeName="kubernetes.io/configmap/a4d955b0-1418-499f-9036-961443f1e130-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557419    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2e85a91-0400-4117-a221-cc7961fd3ba0" volumeName="kubernetes.io/projected/b2e85a91-0400-4117-a221-cc7961fd3ba0-kube-api-access-tktxh" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557427    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea79f848-e58a-463b-8c7f-d6f6dd0125b3" volumeName="kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557437    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="177793bf-7bb1-4b0b-adb6-ba40483ae237" volumeName="kubernetes.io/secret/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557445    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" volumeName="kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-service-ca" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557470    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4d955b0-1418-499f-9036-961443f1e130" volumeName="kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-client-ca-bundle" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557478    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0af7ee9f-95f0-4632-989a-2beaf0fba3e3" volumeName="kubernetes.io/projected/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-kube-api-access-lclzk" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557486    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16799a99-285f-40d5-acbb-a8fbe0d7a2d9" volumeName="kubernetes.io/secret/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-prometheus-operator-tls" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557494    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cb392b3-c5a6-4ae1-af06-82f167ed3c27" volumeName="kubernetes.io/empty-dir/6cb392b3-c5a6-4ae1-af06-82f167ed3c27-tmp" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557502    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" volumeName="kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-serving-cert" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557510    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da297bce-ca72-4dea-b02c-bcda6f401ebe" volumeName="kubernetes.io/secret/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557523    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0af7ee9f-95f0-4632-989a-2beaf0fba3e3" volumeName="kubernetes.io/secret/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557534    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16799a99-285f-40d5-acbb-a8fbe0d7a2d9" volumeName="kubernetes.io/projected/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-kube-api-access-22m44" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557555    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20dcc2be-01c6-4294-bce6-3b9a77f0d3d9" volumeName="kubernetes.io/configmap/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-iptables-alerter-script" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557567    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="761f94e4-2681-4253-abb6-898d5f67aa98" volumeName="kubernetes.io/projected/761f94e4-2681-4253-abb6-898d5f67aa98-kube-api-access-wjbgf" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557581    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35c8150-ce21-4660-ad5d-e5cc1fce684b" volumeName="kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557592    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da297bce-ca72-4dea-b02c-bcda6f401ebe" volumeName="kubernetes.io/configmap/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557600    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0077be45-ce2b-46f0-8fe4-e881254f6386" volumeName="kubernetes.io/secret/0077be45-ce2b-46f0-8fe4-e881254f6386-tls-certificates" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557608    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2e85a91-0400-4117-a221-cc7961fd3ba0" volumeName="kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557617    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" volumeName="kubernetes.io/empty-dir/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-tmp" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557625    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0af7ee9f-95f0-4632-989a-2beaf0fba3e3" volumeName="kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovnkube-config" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557633    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e387f3b0-3991-4570-895b-8b59c01c8f36" volumeName="kubernetes.io/projected/e387f3b0-3991-4570-895b-8b59c01c8f36-kube-api-access-9695l" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557651    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="177793bf-7bb1-4b0b-adb6-ba40483ae237" volumeName="kubernetes.io/secret/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-tls" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557672    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17bd527b-9dd9-4fc6-86b3-57b5456c5a6b" volumeName="kubernetes.io/configmap/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b-konnectivity-ca" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557683    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fdb73b3-0430-4d0c-a813-50a1beebdb12" volumeName="kubernetes.io/projected/8fdb73b3-0430-4d0c-a813-50a1beebdb12-kube-api-access-sf4cc" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557697    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4d955b0-1418-499f-9036-961443f1e130" volumeName="kubernetes.io/configmap/a4d955b0-1418-499f-9036-961443f1e130-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557710    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4d955b0-1418-499f-9036-961443f1e130" volumeName="kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557722    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2e85a91-0400-4117-a221-cc7961fd3ba0" volumeName="kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557731    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9c7f8e8-6c5e-4f69-bede-8e89d342d02a" volumeName="kubernetes.io/projected/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-kube-api-access-nkbch" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557738    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2941755b-c963-4514-a42a-c918a8283273" volumeName="kubernetes.io/secret/2941755b-c963-4514-a42a-c918a8283273-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557745    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cd79b48-c38e-4822-86b5-4b7c3c858fb6" volumeName="kubernetes.io/projected/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-kube-api-access-rnds5" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557754    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="761f94e4-2681-4253-abb6-898d5f67aa98" volumeName="kubernetes.io/secret/761f94e4-2681-4253-abb6-898d5f67aa98-metrics-tls" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557761    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4d955b0-1418-499f-9036-961443f1e130" volumeName="kubernetes.io/projected/a4d955b0-1418-499f-9036-961443f1e130-kube-api-access-n965n" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557768    5415 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da297bce-ca72-4dea-b02c-bcda6f401ebe" volumeName="kubernetes.io/empty-dir/da297bce-ca72-4dea-b02c-bcda6f401ebe-volume-directive-shadow" seLinuxMountContext=""
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557776    5415 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 03:07:38.557830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.557782    5415 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 03:07:38.596539 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.596423    5415 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 03:07:38.597898 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.597842    5415 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 03:07:38.597898 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.597870    5415 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 03:07:38.598092 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.597916    5415 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 03:07:38.598092 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:07:38.597961    5415 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 03:07:38.614657 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.614616    5415 generic.go:356] "Generic (PLEG): container finished" podID="b2e85a91-0400-4117-a221-cc7961fd3ba0" containerID="e0c4df9401b0c678c4ee95c11ccc555c0ec12aaed3560c69982d9846c544a268" exitCode=0
Oct 31 03:07:38.614657 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.614657    5415 generic.go:356] "Generic (PLEG): container finished" podID="b2e85a91-0400-4117-a221-cc7961fd3ba0" containerID="182b1b72d287d8af70dd2f53d6758ab84f85671487ef9b8c995fa91259983248" exitCode=0
Oct 31 03:07:38.615042 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.614669    5415 generic.go:356] "Generic (PLEG): container finished" podID="b2e85a91-0400-4117-a221-cc7961fd3ba0" containerID="2910743bf264e16e55f702a7ffcf56b3c1d737e8b89249f387feace41dd2747f" exitCode=0
Oct 31 03:07:38.615042 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.614680    5415 generic.go:356] "Generic (PLEG): container finished" podID="b2e85a91-0400-4117-a221-cc7961fd3ba0" containerID="73dde071cd64b0d743fc742b089fcd95ed26b87008c5763123df758fd23a86d1" exitCode=0
Oct 31 03:07:38.615042 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.614688    5415 generic.go:356] "Generic (PLEG): container finished" podID="b2e85a91-0400-4117-a221-cc7961fd3ba0" containerID="ca65a0863a030e35121881af8cc9cdd6f28c6c4881a4b10fa85e175ebb28ca6a" exitCode=0
Oct 31 03:07:38.615042 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.614698    5415 generic.go:356] "Generic (PLEG): container finished" podID="b2e85a91-0400-4117-a221-cc7961fd3ba0" containerID="8adf7fe9742b3c591bc35a2ca447f55415a078c6f0e495b6addda94645cabaeb" exitCode=0
Oct 31 03:07:38.624931 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.622475    5415 generic.go:356] "Generic (PLEG): container finished" podID="177793bf-7bb1-4b0b-adb6-ba40483ae237" containerID="3c6eeb53a30fcd8f831d7764f8856a2f49cb8d5647d339ddc0480ee7caca66df" exitCode=0
Oct 31 03:07:38.651822 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:07:38.651787    5415 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-164.us-west-2.compute.internal\" not found"
Oct 31 03:07:38.685921 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.680967    5415 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:07:38.685921 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.681976    5415 generic.go:356] "Generic (PLEG): container finished" podID="41bbbd7b4d4d448be0c950dd63a0f8ce" containerID="d902b90358cd0471f6e43bbd45900acbb62c343711c3d7d585ec9083e368d245" exitCode=0
Oct 31 03:07:38.698051 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:07:38.698018    5415 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 03:07:38.703647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.703619    5415 manager.go:324] Recovery completed
Oct 31 03:07:38.728615 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.728582    5415 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:07:38.728776 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.728590    5415 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:07:38.733177 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.733145    5415 cpu_manager.go:226] "Starting" policy="none"
Oct 31 03:07:38.733177 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.733169    5415 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 03:07:38.733420 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.733191    5415 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 03:07:38.733420 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.733330    5415 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 03:07:38.733420 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.733341    5415 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 03:07:38.733420 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.733360    5415 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 03:07:38.733420 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.733366    5415 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 03:07:38.733420 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.733371    5415 policy_none.go:50] "Start"
Oct 31 03:07:38.733420 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.733380    5415 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 03:07:38.733420 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.733402    5415 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 03:07:38.734869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.734843    5415 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 03:07:38.734869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.734869    5415 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 03:07:38.735025 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.734901    5415 policy_none.go:44] "Start"
Oct 31 03:07:38.741085 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.740554    5415 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 03:07:38.747577 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.747556    5415 manager.go:341] "Starting Device Plugin manager"
Oct 31 03:07:38.747694 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:07:38.747594    5415 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 03:07:38.747694 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.747607    5415 server.go:87] "Starting device plugin registration server"
Oct 31 03:07:38.747853 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.747774    5415 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 03:07:38.747853 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.747787    5415 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 03:07:38.747973 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.747906    5415 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 03:07:38.748079 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.748039    5415 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 03:07:38.748193 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.748086    5415 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 03:07:38.748193 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.748093    5415 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 03:07:38.748308 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.748142    5415 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:07:38.748073855Z","UUID":"e0021e80-ca5a-454b-addb-00c9435b0bb3","Handler":null,"Name":"","Endpoint":""}
Oct 31 03:07:38.749797 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:07:38.749775    5415 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 03:07:38.750232 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.750216    5415 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 03:07:38.750313 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.750241    5415 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 03:07:38.773377 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.773348    5415 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 03:07:38.821398 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.821355    5415 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:07:38.848622 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.848585    5415 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:07:38.850660 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.850630    5415 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:07:38.850815 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.850672    5415 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:07:38.850815 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.850689    5415 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-164.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:07:38.850815 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.850746    5415 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:38.898824 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.898692    5415 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"]
Oct 31 03:07:38.899590 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.899541    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal" event={"ID":"41bbbd7b4d4d448be0c950dd63a0f8ce","Type":"ContainerStarted","Data":"78de8ed21133c98abb9e253f8616398f413c92748eebc5e316beeb74687f9927"}
Oct 31 03:07:38.899590 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.899587    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal" event={"ID":"41bbbd7b4d4d448be0c950dd63a0f8ce","Type":"ContainerDied","Data":"d902b90358cd0471f6e43bbd45900acbb62c343711c3d7d585ec9083e368d245"}
Oct 31 03:07:38.899743 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.899607    5415 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:38.899743 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.899608    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal" event={"ID":"41bbbd7b4d4d448be0c950dd63a0f8ce","Type":"ContainerStarted","Data":"482d8994a27be3a92c38394df337d8d0450d366ebd3bd3a47535b14a4d4be0da"}
Oct 31 03:07:38.899809 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.899752    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal" event={"ID":"d510a50aad744fb4754386c4136c4a71","Type":"ContainerStarted","Data":"d3b4e3d5797ab6c6cf1652b19ff83f3885cd9cba23e721ca31528cd0b2ed7a08"}
Oct 31 03:07:38.899809 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.899786    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal" event={"ID":"d510a50aad744fb4754386c4136c4a71","Type":"ContainerStarted","Data":"8dda0ac8387681a083e0ccbaa2c30e7472baba88429468c1ce6d7bde63099e5a"}
Oct 31 03:07:38.899871 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.899789    5415 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:38.956076 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.956047    5415 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:07:38.956188 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:07:38.956099    5415 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:38.956188 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.956146    5415 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:38.956245 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.956209    5415 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:38.956598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.956584    5415 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:07:38.956676 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:07:38.956618    5415 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:38.974084 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.974048    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41bbbd7b4d4d448be0c950dd63a0f8ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"41bbbd7b4d4d448be0c950dd63a0f8ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:38.974084 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.974085    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41bbbd7b4d4d448be0c950dd63a0f8ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"41bbbd7b4d4d448be0c950dd63a0f8ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:38.974192 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:38.974104    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d510a50aad744fb4754386c4136c4a71-config\") pod \"kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"d510a50aad744fb4754386c4136c4a71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:39.074271 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.074235    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41bbbd7b4d4d448be0c950dd63a0f8ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"41bbbd7b4d4d448be0c950dd63a0f8ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:39.074271 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.074276    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41bbbd7b4d4d448be0c950dd63a0f8ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"41bbbd7b4d4d448be0c950dd63a0f8ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:39.074478 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.074327    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41bbbd7b4d4d448be0c950dd63a0f8ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"41bbbd7b4d4d448be0c950dd63a0f8ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:39.074478 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.074331    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41bbbd7b4d4d448be0c950dd63a0f8ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"41bbbd7b4d4d448be0c950dd63a0f8ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:39.074478 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.074371    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d510a50aad744fb4754386c4136c4a71-config\") pod \"kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"d510a50aad744fb4754386c4136c4a71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:39.074478 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.074401    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d510a50aad744fb4754386c4136c4a71-config\") pod \"kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal\" (UID: \"d510a50aad744fb4754386c4136c4a71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:39.525080 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.525042    5415 apiserver.go:52] "Watching apiserver"
Oct 31 03:07:39.709950 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.709921    5415 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:39.710093 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.710027    5415 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:39.779985 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.779895    5415 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:07:39.779985 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:07:39.779966    5415 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:39.780233 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.779899    5415 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:07:39.780233 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:07:39.780053    5415 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal"
Oct 31 03:07:39.780233 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.780168    5415 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 03:07:39.780894 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.780857    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-nv7ws","openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr","openshift-monitoring/node-exporter-2cjh2","openshift-console/downloads-6dd789c8b7-7rr4c","openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq","openshift-multus/network-metrics-daemon-sl5hn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-164.us-west-2.compute.internal","openshift-multus/multus-68wpw","kube-system/konnectivity-agent-x4wzm","openshift-dns/dns-default-2bqmj","openshift-network-diagnostics/network-check-target-mr7hm","openshift-console/console-74c979ccc9-r9gs5","openshift-insights/insights-runtime-extractor-8452g","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk","kube-system/global-pull-secret-syncer-ldcbk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj","openshift-cluster-node-tuning-operator/tuned-9775g","openshift-dns/node-resolver-fbw9q","kube-system/kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-dkpt8","openshift-monitoring/metrics-server-7887dcbc55-2plp6","openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58","openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj","openshift-image-registry/node-ca-bzwqk","openshift-ingress-canary/ingress-canary-4kbs4","openshift-multus/multus-additional-cni-plugins-dh7cw"]
Oct 31 03:07:39.832088 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.832051    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z58pm\""
Oct 31 03:07:39.833058 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.833033    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 03:07:39.833146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.833033    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lsfgs\""
Oct 31 03:07:39.833146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.833068    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 03:07:39.833146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.833045    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 03:07:39.833281 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.833265    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ffsgr\""
Oct 31 03:07:39.834159 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.834139    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 03:07:39.834378 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.834360    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 03:07:39.834445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.834385    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 03:07:39.834445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.834411    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 03:07:39.834445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.834412    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 03:07:39.834445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.834387    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 03:07:39.834445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.834441    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 03:07:39.834635 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.834410    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 03:07:39.834635 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.834622    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 03:07:39.854384 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.854355    5415 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 03:07:39.879101 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879065    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:07:39.879193 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879108    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.879193 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879129    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.879193 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879170    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-daemon-config\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.879367 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879223    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-kubernetes\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.879367 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879267    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761f94e4-2681-4253-abb6-898d5f67aa98-tmp-dir\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:39.879367 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879288    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.879367 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879342    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-cni-multus\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.879533 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879384    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.879533 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879391    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.879533 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879399    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-sys\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.879533 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879427    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fbec67c-68ae-48f9-b780-bff7931da269-metrics-certs\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:07:39.879533 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879429    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:39.879533 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879428    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761f94e4-2681-4253-abb6-898d5f67aa98-tmp-dir\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:39.879533 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879466    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22m44\" (UniqueName: \"kubernetes.io/projected/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-kube-api-access-22m44\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:39.879722 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879537    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-tls\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.879722 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879616    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4cc\" (UniqueName: \"kubernetes.io/projected/8fdb73b3-0430-4d0c-a813-50a1beebdb12-kube-api-access-sf4cc\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.879779 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879726    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qnj\" (UniqueName: \"kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj\") pod \"network-check-target-mr7hm\" (UID: \"ea79f848-e58a-463b-8c7f-d6f6dd0125b3\") " pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:07:39.879779 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879756    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:39.879856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879778    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-iptables-alerter-script\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:07:39.879856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879807    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-k8s-cni-cncf-io\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.879856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879830    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-var-lib-kubelet\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.879856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879846    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-config\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.880051 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879863    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-host\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:07:39.880051 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.879919    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-kubelet\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.880051 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880012    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d35c8150-ce21-4660-ad5d-e5cc1fce684b-kubelet-config\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:39.880051 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880041    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-cni-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.880164 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880064    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-netns\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.880164 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880091    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysctl-conf\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.880164 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880110    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zvxt\" (UniqueName: \"kubernetes.io/projected/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-kube-api-access-6zvxt\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.880164 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880124    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/761f94e4-2681-4253-abb6-898d5f67aa98-metrics-tls\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:39.880164 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880135    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35c8150-ce21-4660-ad5d-e5cc1fce684b-original-pull-secret\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:39.880296 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880161    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-system-cni-dir\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.880296 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880241    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/382f0f50-a8e4-463d-a64c-710cfffdc963-cni-binary-copy\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.880349 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880320    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-socket-dir-parent\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.880381 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880372    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a4d955b0-1418-499f-9036-961443f1e130-audit-log\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.880456 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880439    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:39.880487 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880468    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a4d955b0-1418-499f-9036-961443f1e130-audit-log\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.880487 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880472    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/382f0f50-a8e4-463d-a64c-710cfffdc963-cni-binary-copy\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.880539 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880476    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0077be45-ce2b-46f0-8fe4-e881254f6386-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wtgvk\" (UID: \"0077be45-ce2b-46f0-8fe4-e881254f6386\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk"
Oct 31 03:07:39.880539 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880523    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-root\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.880597 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880557    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lclzk\" (UniqueName: \"kubernetes.io/projected/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-kube-api-access-lclzk\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.880597 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880574    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjbgf\" (UniqueName: \"kubernetes.io/projected/761f94e4-2681-4253-abb6-898d5f67aa98-kube-api-access-wjbgf\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:39.880653 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880608    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d78tf\" (UniqueName: \"kubernetes.io/projected/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-kube-api-access-d78tf\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:07:39.880653 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880630    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9695l\" (UniqueName: \"kubernetes.io/projected/e387f3b0-3991-4570-895b-8b59c01c8f36-kube-api-access-9695l\") pod \"ingress-canary-4kbs4\" (UID: \"e387f3b0-3991-4570-895b-8b59c01c8f36\") " pod="openshift-ingress-canary/ingress-canary-4kbs4"
Oct 31 03:07:39.880653 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880648    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-modprobe-d\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.880737 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880678    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysconfig\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.880737 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880718    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-systemd\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.880789 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880773    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-cni-netd\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.880894 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880864    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.880952 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880921    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-systemd\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.880983 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880966    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5j7c\" (UniqueName: \"kubernetes.io/projected/6cb392b3-c5a6-4ae1-af06-82f167ed3c27-kube-api-access-k5j7c\") pod \"downloads-6dd789c8b7-7rr4c\" (UID: \"6cb392b3-c5a6-4ae1-af06-82f167ed3c27\") " pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:07:39.881011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.880990    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.881040 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881024    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-registration-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.881072 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881051    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b-agent-certs\") pod \"konnectivity-agent-x4wzm\" (UID: \"17bd527b-9dd9-4fc6-86b3-57b5456c5a6b\") " pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:07:39.881112 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881086    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnpf\" (UniqueName: \"kubernetes.io/projected/0fbec67c-68ae-48f9-b780-bff7931da269-kube-api-access-4dnpf\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:07:39.881143 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881123    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-crio-socket\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:39.881171 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881150    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2941755b-c963-4514-a42a-c918a8283273-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:39.881200 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881180    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-run\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.881231 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881208    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-oauth-serving-cert\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.881309 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881293    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.881339 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881320    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-oauth-config\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.881367 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881337    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-run-netns\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.881405 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881380    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2941755b-c963-4514-a42a-c918a8283273-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:39.881438 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881407    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-service-ca\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.881468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881438    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-tmp\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.881522 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881506    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/761f94e4-2681-4253-abb6-898d5f67aa98-config-volume\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:39.881551 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881539    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-wtmp\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.881580 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881507    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-tmp\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.881580 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881564    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-hostroot\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.881635 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881586    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknn7\" (UniqueName: \"kubernetes.io/projected/382f0f50-a8e4-463d-a64c-710cfffdc963-kube-api-access-tknn7\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.881635 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881603    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-data-volume\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:39.881635 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881621    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-systemd-units\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.881713 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881644    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.881713 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881668    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-cnibin\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.881713 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881670    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-data-volume\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:39.881713 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881684    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-serviceca\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:07:39.881816 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881799    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovn-node-metrics-cert\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.881870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881856    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8c5k\" (UniqueName: \"kubernetes.io/projected/fad83c1a-3d12-44d3-9164-4c015d044f3e-kube-api-access-r8c5k\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.881950 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881935    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-conf-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.881989 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881963    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4d955b0-1418-499f-9036-961443f1e130-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.881989 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.881979    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cb392b3-c5a6-4ae1-af06-82f167ed3c27-tmp\") pod \"downloads-6dd789c8b7-7rr4c\" (UID: \"6cb392b3-c5a6-4ae1-af06-82f167ed3c27\") " pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:07:39.882064 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882000    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-slash\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.882118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882058    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-socket-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.882118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882082    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovn-node-metrics-cert\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.882118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882101    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cb392b3-c5a6-4ae1-af06-82f167ed3c27-tmp\") pod \"downloads-6dd789c8b7-7rr4c\" (UID: \"6cb392b3-c5a6-4ae1-af06-82f167ed3c27\") " pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:07:39.882118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882103    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:39.882291 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882143    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-serving-cert\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.882291 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882233    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovnkube-config\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.882360 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882326    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-textfile\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.882360 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882346    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-secret-metrics-server-tls\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.882423 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882363    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-lib-modules\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.882423 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882373    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-textfile\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.882423 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882380    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-env-overrides\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.882423 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882323    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-serviceca\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:07:39.882423 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882397    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-tmp-dir\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:07:39.882560 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882477    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-multus-certs\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.882560 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882508    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-tmp-dir\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:07:39.882560 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882518    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da297bce-ca72-4dea-b02c-bcda6f401ebe-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:39.882560 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882552    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8bd5\" (UniqueName: \"kubernetes.io/projected/a18739ba-0914-4b52-bcda-4a5fafe22225-kube-api-access-g8bd5\") pod \"monitoring-plugin-77f4f76594-p2kxj\" (UID: \"a18739ba-0914-4b52-bcda-4a5fafe22225\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:39.882674 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882571    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-sys-fs\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.882674 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882588    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-host-slash\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:07:39.882674 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882605    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-cni-bin\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.882674 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882624    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-client-ca-bundle\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.882777 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882732    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-tuned\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.882777 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882764    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.882830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882791    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnds5\" (UniqueName: \"kubernetes.io/projected/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-kube-api-access-rnds5\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:07:39.882830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882817    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-ovn\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.882830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882823    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-tuned\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.882933 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882850    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-log-socket\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.882933 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882901    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-cni-bin\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.882933 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882924    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:39.883017 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882973    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:39.883017 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.882992    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n965n\" (UniqueName: \"kubernetes.io/projected/a4d955b0-1418-499f-9036-961443f1e130-kube-api-access-n965n\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.883080 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883071    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-device-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.883170 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883151    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d35c8150-ce21-4660-ad5d-e5cc1fce684b-dbus\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:39.883211 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883198    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-cni-binary-copy\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.883247 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883222    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tktxh\" (UniqueName: \"kubernetes.io/projected/b2e85a91-0400-4117-a221-cc7961fd3ba0-kube-api-access-tktxh\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.883247 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883241    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-etc-kubernetes\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.883300 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883257    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-hosts-file\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:07:39.883300 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883289    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2941755b-c963-4514-a42a-c918a8283273-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:39.883413 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883401    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m5bp\" (UniqueName: \"kubernetes.io/projected/2941755b-c963-4514-a42a-c918a8283273-kube-api-access-9m5bp\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:39.883477 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883459    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2e85a91-0400-4117-a221-cc7961fd3ba0-cni-binary-copy\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.883507 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883480    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:39.883507 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883500    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-cnibin\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.883558 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883516    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e387f3b0-3991-4570-895b-8b59c01c8f36-cert\") pod \"ingress-canary-4kbs4\" (UID: \"e387f3b0-3991-4570-895b-8b59c01c8f36\") " pod="openshift-ingress-canary/ingress-canary-4kbs4"
Oct 31 03:07:39.883558 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883535    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-host\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.883613 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883563    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fdb73b3-0430-4d0c-a813-50a1beebdb12-tmp\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.883613 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883585    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-node-log\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.883613 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883605    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovnkube-script-lib\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.883693 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883654    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:39.883693 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883671    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fdb73b3-0430-4d0c-a813-50a1beebdb12-tmp\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.883749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883693    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a18739ba-0914-4b52-bcda-4a5fafe22225-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p2kxj\" (UID: \"a18739ba-0914-4b52-bcda-4a5fafe22225\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:39.883749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883725    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-kubelet\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.883809 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883749    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4q5r\" (UniqueName: \"kubernetes.io/projected/177793bf-7bb1-4b0b-adb6-ba40483ae237-kube-api-access-b4q5r\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.883809 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883798    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/da297bce-ca72-4dea-b02c-bcda6f401ebe-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:39.883865 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883822    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-etc-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.883865 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883840    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b-konnectivity-ca\") pod \"konnectivity-agent-x4wzm\" (UID: \"17bd527b-9dd9-4fc6-86b3-57b5456c5a6b\") " pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:07:39.883865 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883859    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-sys\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.883970 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883903    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177793bf-7bb1-4b0b-adb6-ba40483ae237-metrics-client-ca\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.883970 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883924    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlpv\" (UniqueName: \"kubernetes.io/projected/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-api-access-xvlpv\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:39.883970 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883947    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/da297bce-ca72-4dea-b02c-bcda6f401ebe-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:39.884050 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.883972    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-secret-metrics-server-client-certs\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.884050 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.884013    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-var-lib-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.884103 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.884043    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.884103 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.884077    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:39.884161 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.884111    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkbch\" (UniqueName: \"kubernetes.io/projected/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-kube-api-access-nkbch\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:07:39.884199 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.884184    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-system-cni-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.884228 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.884220    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2fzj\" (UniqueName: \"kubernetes.io/projected/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-kube-api-access-n2fzj\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:39.884259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.884247    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a4d955b0-1418-499f-9036-961443f1e130-metrics-server-audit-profiles\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.884292 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.884271    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysctl-d\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.884324 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.884299    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.916798 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.916763    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 03:07:39.916990 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.916814    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 03:07:39.916990 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.916893    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 03:07:39.916990 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.916917    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:07:39.916990 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.916934    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 03:07:39.916990 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.916980    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 03:07:39.917263 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.917042    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 03:07:39.917263 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.917070    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 03:07:39.917263 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.917075    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 03:07:39.917263 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.917087    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 03:07:39.917263 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.917138    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 03:07:39.917485 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.917331    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 03:07:39.917485 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.917364    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:07:39.917485 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.917371    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 03:07:39.917485 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.917384    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 03:07:39.917485 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.917443    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 03:07:39.917485 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.917450    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 03:07:39.917658 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.917645    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 03:07:39.917980 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.917968    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 03:07:39.918145 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918117    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 03:07:39.918145 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918112    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 03:07:39.918145 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918121    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 03:07:39.918145 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918126    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 03:07:39.918299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918159    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 03:07:39.918299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918184    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 03:07:39.918509 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918485    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 03:07:39.918589 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918564    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 03:07:39.918589 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918580    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 03:07:39.918653 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918607    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 03:07:39.918714 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918702    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 03:07:39.918766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918756    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 03:07:39.918800 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918789    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 03:07:39.918852 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918842    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 03:07:39.918901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918888    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 03:07:39.918968 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.918957    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 03:07:39.919687 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.919673    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-daemon-config\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.920814 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.920792    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-config\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.920869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.920813    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-iptables-alerter-script\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:07:39.921906 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.921864    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/761f94e4-2681-4253-abb6-898d5f67aa98-config-volume\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:39.921974 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.921907    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.921974 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.921950    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-service-ca\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.921974 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.921955    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-oauth-serving-cert\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.923118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.923099    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-env-overrides\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.923156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.923126    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da297bce-ca72-4dea-b02c-bcda6f401ebe-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:39.923156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.923134    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovnkube-config\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.923217 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.923157    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4d955b0-1418-499f-9036-961443f1e130-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.923273 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.923259    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:39.924295 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.924275    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2941755b-c963-4514-a42a-c918a8283273-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:39.924366 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.924293    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177793bf-7bb1-4b0b-adb6-ba40483ae237-metrics-client-ca\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.924366 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.924283    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:39.924366 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.924331    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-ovnkube-script-lib\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.924366 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.924277    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:39.924366 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.924330    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b-konnectivity-ca\") pod \"konnectivity-agent-x4wzm\" (UID: \"17bd527b-9dd9-4fc6-86b3-57b5456c5a6b\") " pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:07:39.925582 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.925566    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a4d955b0-1418-499f-9036-961443f1e130-metrics-server-audit-profiles\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.933300 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.933280    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22m44\" (UniqueName: \"kubernetes.io/projected/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-kube-api-access-22m44\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:39.933863 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.933841    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lclzk\" (UniqueName: \"kubernetes.io/projected/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-kube-api-access-lclzk\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.933925 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.933842    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4cc\" (UniqueName: \"kubernetes.io/projected/8fdb73b3-0430-4d0c-a813-50a1beebdb12-kube-api-access-sf4cc\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.933990 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.933974    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9695l\" (UniqueName: \"kubernetes.io/projected/e387f3b0-3991-4570-895b-8b59c01c8f36-kube-api-access-9695l\") pod \"ingress-canary-4kbs4\" (UID: \"e387f3b0-3991-4570-895b-8b59c01c8f36\") " pod="openshift-ingress-canary/ingress-canary-4kbs4"
Oct 31 03:07:39.934027 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.934003    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zvxt\" (UniqueName: \"kubernetes.io/projected/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-kube-api-access-6zvxt\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.934091 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.934077    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8c5k\" (UniqueName: \"kubernetes.io/projected/fad83c1a-3d12-44d3-9164-4c015d044f3e-kube-api-access-r8c5k\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.934954 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.934863    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5j7c\" (UniqueName: \"kubernetes.io/projected/6cb392b3-c5a6-4ae1-af06-82f167ed3c27-kube-api-access-k5j7c\") pod \"downloads-6dd789c8b7-7rr4c\" (UID: \"6cb392b3-c5a6-4ae1-af06-82f167ed3c27\") " pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:07:39.935141 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.935095    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tknn7\" (UniqueName: \"kubernetes.io/projected/382f0f50-a8e4-463d-a64c-710cfffdc963-kube-api-access-tknn7\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.935191 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.935120    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnpf\" (UniqueName: \"kubernetes.io/projected/0fbec67c-68ae-48f9-b780-bff7931da269-kube-api-access-4dnpf\") pod \"network-metrics-daemon-sl5hn\" (UID: \"0fbec67c-68ae-48f9-b780-bff7931da269\") " pod="openshift-multus/network-metrics-daemon-sl5hn"
Oct 31 03:07:39.935289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.935274    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjbgf\" (UniqueName: \"kubernetes.io/projected/761f94e4-2681-4253-abb6-898d5f67aa98-kube-api-access-wjbgf\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:39.935633 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.935616    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94qnj\" (UniqueName: \"kubernetes.io/projected/ea79f848-e58a-463b-8c7f-d6f6dd0125b3-kube-api-access-94qnj\") pod \"network-check-target-mr7hm\" (UID: \"ea79f848-e58a-463b-8c7f-d6f6dd0125b3\") " pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:07:39.936380 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.936353    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 03:07:39.936511 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.936498    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-79t2s\""
Oct 31 03:07:39.936690 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.936666    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 03:07:39.936893 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.936860    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 03:07:39.936893 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.936871    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sc79n\""
Oct 31 03:07:39.937028 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.936962    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 03:07:39.941122 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.941095    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/761f94e4-2681-4253-abb6-898d5f67aa98-metrics-tls\") pod \"dns-default-2bqmj\" (UID: \"761f94e4-2681-4253-abb6-898d5f67aa98\") " pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:39.941235 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.941129    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 03:07:39.941235 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.941107    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lfdkn\""
Oct 31 03:07:39.941235 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.941152    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 03:07:39.941331 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.941291    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k74vl\""
Oct 31 03:07:39.941493 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.941465    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2941755b-c963-4514-a42a-c918a8283273-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:39.941649 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.941635    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.941863 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.941850    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fb2gc\""
Oct 31 03:07:39.942189 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.942172    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 03:07:39.942243 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.942227    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-87zff\""
Oct 31 03:07:39.942370 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.942357    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:39.943399 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.943385    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:39.943399 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.943394    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-secret-metrics-server-tls\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.950613 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.950595    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:39.964147 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964117    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-47gs9\""
Oct 31 03:07:39.964298 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964130    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-89z7n\""
Oct 31 03:07:39.964298 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964192    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-57bwv\""
Oct 31 03:07:39.964298 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964216    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gtdck\""
Oct 31 03:07:39.964448 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964334    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 03:07:39.964448 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964335    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4744q\""
Oct 31 03:07:39.964448 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964362    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 03:07:39.964448 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964397    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5quaqkm7oduoe\""
Oct 31 03:07:39.964448 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964431    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 03:07:39.964448 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964436    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 03:07:39.964699 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964548    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 03:07:39.964699 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964594    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 03:07:39.964699 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964626    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 03:07:39.964699 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964685    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2wckk\""
Oct 31 03:07:39.964902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964709    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-95c6m\""
Oct 31 03:07:39.964902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964742    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m6gdv\""
Oct 31 03:07:39.964902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964744    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 03:07:39.964902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964773    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6df8z\""
Oct 31 03:07:39.964902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964834    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dzlls\""
Oct 31 03:07:39.964902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964840    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l8z7f\""
Oct 31 03:07:39.964902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964862    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 03:07:39.965230 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.964991    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 03:07:39.965230 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.965071    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 03:07:39.965230 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.965080    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 03:07:39.965230 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.965162    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8l9rs\""
Oct 31 03:07:39.965518 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.965500    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9g6gp\""
Oct 31 03:07:39.965648 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.965629    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bdmwz\""
Oct 31 03:07:39.965648 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.965642    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wjqqx\""
Oct 31 03:07:39.968468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.966345    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4q5r\" (UniqueName: \"kubernetes.io/projected/177793bf-7bb1-4b0b-adb6-ba40483ae237-kube-api-access-b4q5r\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.968468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.967570    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d78tf\" (UniqueName: \"kubernetes.io/projected/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-kube-api-access-d78tf\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:07:39.968468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.967635    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8bd5\" (UniqueName: \"kubernetes.io/projected/a18739ba-0914-4b52-bcda-4a5fafe22225-kube-api-access-g8bd5\") pod \"monitoring-plugin-77f4f76594-p2kxj\" (UID: \"a18739ba-0914-4b52-bcda-4a5fafe22225\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:39.968468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.968352    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n965n\" (UniqueName: \"kubernetes.io/projected/a4d955b0-1418-499f-9036-961443f1e130-kube-api-access-n965n\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.968468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.968446    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlpv\" (UniqueName: \"kubernetes.io/projected/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-api-access-xvlpv\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:39.969365 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.968677    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkbch\" (UniqueName: \"kubernetes.io/projected/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-kube-api-access-nkbch\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:07:39.969365 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.968679    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2fzj\" (UniqueName: \"kubernetes.io/projected/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-kube-api-access-n2fzj\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:39.969365 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.968894    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnds5\" (UniqueName: \"kubernetes.io/projected/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-kube-api-access-rnds5\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:07:39.969365 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.969099    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tktxh\" (UniqueName: \"kubernetes.io/projected/b2e85a91-0400-4117-a221-cc7961fd3ba0-kube-api-access-tktxh\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.969365 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.969202    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m5bp\" (UniqueName: \"kubernetes.io/projected/2941755b-c963-4514-a42a-c918a8283273-kube-api-access-9m5bp\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:39.970759 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.970732    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da297bce-ca72-4dea-b02c-bcda6f401ebe-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4kp58\" (UID: \"da297bce-ca72-4dea-b02c-bcda6f401ebe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4kp58"
Oct 31 03:07:39.970759 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.970739    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-tls\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.970888 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.970784    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0077be45-ce2b-46f0-8fe4-e881254f6386-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wtgvk\" (UID: \"0077be45-ce2b-46f0-8fe4-e881254f6386\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk"
Oct 31 03:07:39.971976 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.971946    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-oauth-config\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.973709 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.971991    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17bd527b-9dd9-4fc6-86b3-57b5456c5a6b-agent-certs\") pod \"konnectivity-agent-x4wzm\" (UID: \"17bd527b-9dd9-4fc6-86b3-57b5456c5a6b\") " pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:07:39.973709 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.972049    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2941755b-c963-4514-a42a-c918a8283273-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mbdfr\" (UID: \"2941755b-c963-4514-a42a-c918a8283273\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mbdfr"
Oct 31 03:07:39.973900 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.973827    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e387f3b0-3991-4570-895b-8b59c01c8f36-cert\") pod \"ingress-canary-4kbs4\" (UID: \"e387f3b0-3991-4570-895b-8b59c01c8f36\") " pod="openshift-ingress-canary/ingress-canary-4kbs4"
Oct 31 03:07:39.973956 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.973924    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-serving-cert\") pod \"console-74c979ccc9-r9gs5\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") " pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:39.974006 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.973954    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a18739ba-0914-4b52-bcda-4a5fafe22225-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p2kxj\" (UID: \"a18739ba-0914-4b52-bcda-4a5fafe22225\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:39.974006 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.973972    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-client-ca-bundle\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.975064 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.975049    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16799a99-285f-40d5-acbb-a8fbe0d7a2d9-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9pbqq\" (UID: \"16799a99-285f-40d5-acbb-a8fbe0d7a2d9\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9pbqq"
Oct 31 03:07:39.975141 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.975127    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a4d955b0-1418-499f-9036-961443f1e130-secret-metrics-server-client-certs\") pod \"metrics-server-7887dcbc55-2plp6\" (UID: \"a4d955b0-1418-499f-9036-961443f1e130\") " pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:39.985693 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985661    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-cni-bin\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.985855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985706    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.985855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985729    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-ovn\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.985855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985749    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-log-socket\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.985855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985785    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-cni-bin\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.985855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985782    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.985855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985792    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-ovn\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.985855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985817    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-log-socket\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.985855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985792    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-cni-bin\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.985855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985837    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-device-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.985855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985832    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-cni-bin\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985867    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d35c8150-ce21-4660-ad5d-e5cc1fce684b-dbus\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985936    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-device-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985937    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d35c8150-ce21-4660-ad5d-e5cc1fce684b-dbus\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985949    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-etc-kubernetes\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985971    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-etc-kubernetes\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.985993    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-hosts-file\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986022    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-cnibin\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986044    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-host\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986112    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9c7f8e8-6c5e-4f69-bede-8e89d342d02a-hosts-file\") pod \"node-resolver-fbw9q\" (UID: \"e9c7f8e8-6c5e-4f69-bede-8e89d342d02a\") " pod="openshift-dns/node-resolver-fbw9q"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986114    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-cnibin\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986128    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-node-log\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986157    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-kubelet\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986177    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-host\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986191    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-etc-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986204    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-kubelet\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986206    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-node-log\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986216    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-sys\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986226    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-etc-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986247    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-var-lib-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986291    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986301    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-sys\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.986353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986346    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-system-cni-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986363    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986379    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysctl-d\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986376    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-var-lib-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986423    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986430    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-system-cni-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986456    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-kubernetes\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986473    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986490    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysctl-d\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986492    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986518    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-kubernetes\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986518    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-cni-multus\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986561    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-sys\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986567    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986572    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-var-lib-cni-multus\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986625    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-k8s-cni-cncf-io\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986652    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-var-lib-kubelet\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986681    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-sys\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986681    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-host\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986724    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-kubelet\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986736    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-k8s-cni-cncf-io\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986749    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d35c8150-ce21-4660-ad5d-e5cc1fce684b-kubelet-config\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986761    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-var-lib-kubelet\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986765    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cd79b48-c38e-4822-86b5-4b7c3c858fb6-host\") pod \"node-ca-bzwqk\" (UID: \"4cd79b48-c38e-4822-86b5-4b7c3c858fb6\") " pod="openshift-image-registry/node-ca-bzwqk"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986771    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-cni-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986789    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-kubelet\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986831    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-netns\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986836    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-cni-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986863    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d35c8150-ce21-4660-ad5d-e5cc1fce684b-kubelet-config\") pod \"global-pull-secret-syncer-ldcbk\" (UID: \"d35c8150-ce21-4660-ad5d-e5cc1fce684b\") " pod="kube-system/global-pull-secret-syncer-ldcbk"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986871    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-netns\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986907    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysctl-conf\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986932    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-system-cni-dir\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986969    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-socket-dir-parent\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986989    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-root\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.986995    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-system-cni-dir\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987005    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-modprobe-d\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987045    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-root\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987046    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysconfig\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987051    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-socket-dir-parent\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987052    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysctl-conf\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987082    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-systemd\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987099    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-sysconfig\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987114    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-cni-netd\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987148    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-systemd\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987152    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-systemd\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987179    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-cni-netd\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987179    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987200    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-systemd\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987155    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-etc-modprobe-d\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987213    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-run-openvswitch\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987201    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-registration-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987257    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-crio-socket\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987273    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-registration-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987309    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-run\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987356    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c88db770-eb3f-4879-8a41-7e4d7d87fe0e-crio-socket\") pod \"insights-runtime-extractor-8452g\" (UID: \"c88db770-eb3f-4879-8a41-7e4d7d87fe0e\") " pod="openshift-insights/insights-runtime-extractor-8452g"
Oct 31 03:07:39.987325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987357    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-run-netns\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987385    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-run\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987391    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-run-netns\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987397    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-wtmp\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987423    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-hostroot\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987451    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-systemd-units\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987483    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987496    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/177793bf-7bb1-4b0b-adb6-ba40483ae237-node-exporter-wtmp\") pod \"node-exporter-2cjh2\" (UID: \"177793bf-7bb1-4b0b-adb6-ba40483ae237\") " pod="openshift-monitoring/node-exporter-2cjh2"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987508    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-cnibin\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987538    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-conf-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987537    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987529    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-systemd-units\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987567    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-slash\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987584    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-hostroot\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987588    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e85a91-0400-4117-a221-cc7961fd3ba0-cnibin\") pod \"multus-additional-cni-plugins-dh7cw\" (UID: \"b2e85a91-0400-4117-a221-cc7961fd3ba0\") " pod="openshift-multus/multus-additional-cni-plugins-dh7cw"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987613    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-multus-conf-dir\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987644    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-socket-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987590    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-socket-dir\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987682    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0af7ee9f-95f0-4632-989a-2beaf0fba3e3-host-slash\") pod \"ovnkube-node-nv7ws\" (UID: \"0af7ee9f-95f0-4632-989a-2beaf0fba3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987684    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-lib-modules\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987731    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-multus-certs\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987753    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-sys-fs\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987758    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fdb73b3-0430-4d0c-a813-50a1beebdb12-lib-modules\") pod \"tuned-9775g\" (UID: \"8fdb73b3-0430-4d0c-a813-50a1beebdb12\") " pod="openshift-cluster-node-tuning-operator/tuned-9775g"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987766    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-host-slash\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987794    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/382f0f50-a8e4-463d-a64c-710cfffdc963-host-run-multus-certs\") pod \"multus-68wpw\" (UID: \"382f0f50-a8e4-463d-a64c-710cfffdc963\") " pod="openshift-multus/multus-68wpw"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987820    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20dcc2be-01c6-4294-bce6-3b9a77f0d3d9-host-slash\") pod \"iptables-alerter-dkpt8\" (UID: \"20dcc2be-01c6-4294-bce6-3b9a77f0d3d9\") " pod="openshift-network-operator/iptables-alerter-dkpt8"
Oct 31 03:07:39.988895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:39.987851    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fad83c1a-3d12-44d3-9164-4c015d044f3e-sys-fs\") pod \"aws-ebs-csi-driver-node-6qfpj\" (UID: \"fad83c1a-3d12-44d3-9164-4c015d044f3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qfpj"
Oct 31 03:07:40.090971 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.090950    5415 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:07:40.624474 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.624440    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 03:07:40.628920 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.628872    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.680289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.679987    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 03:07:40.680289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.680010    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 03:07:40.680289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.680066    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 03:07:40.680289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.680052    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 03:07:40.680289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.680095    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 03:07:40.680289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.679987    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 03:07:40.680289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.680110    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 03:07:40.680289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.680158    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 03:07:40.680289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.680160    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 03:07:40.680289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.680189    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 03:07:40.680289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.680257    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 03:07:40.680928 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.680309    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 03:07:40.680928 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.680383    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k8rng\""
Oct 31 03:07:40.680928 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.680551    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5vkgn06l0rnmt\""
Oct 31 03:07:40.681667 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.681648    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 03:07:40.681950 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.681936    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 03:07:40.699155 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.699118    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.699408 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.699384    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.699547 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.699531    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.699651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.699637    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.699769 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.699752    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.699930 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.699911    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.700055 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.700039    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.700186 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.700164    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.700295 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.700280    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.700435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.700419    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.700564 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.700550    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.700740 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.700711    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.700951 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.700919    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.701072 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.701047    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 03:07:40.701293 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.701072    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.701472 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.701454    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.701584 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.701570    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.701740 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.701708    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.701847 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.701829    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.701946 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.701895    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjhm9\" (UniqueName: \"kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-kube-api-access-zjhm9\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.701946 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.701935    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802317 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802281    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802317 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802319    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802561 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802341    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802561 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802369    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802561 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802396    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802561 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802438    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802561 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802465    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802561 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802529    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802857 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802577    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802857 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802604    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802857 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802642    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802857 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802665    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802857 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802695    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802857 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802727    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802857 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802747    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802857 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802777    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802857 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802804    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjhm9\" (UniqueName: \"kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-kube-api-access-zjhm9\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.802857 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802834    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.803413 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802903    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.803413 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802769    5415 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 03:07:40.803413 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802935    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.803413 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.802938    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.803602 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.803484    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.804092 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.804062    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.804280 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.804247    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.804489 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.804433    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.805920 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.805892    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.806359 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.806334    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.807001 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.806977    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.807478 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.807456    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.808146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.808124    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.808289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.807687    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.808289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.807796    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.808412 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.808006    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.808412 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.807521    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.808412 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.808181    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.808596 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.808576    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.808826 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.808807    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.809129 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.809110    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.809866 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.809846    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.860713 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.860683    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjhm9\" (UniqueName: \"kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-kube-api-access-zjhm9\") pod \"prometheus-k8s-0\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:40.951838 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:40.951740    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:41.372057 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:41.372021    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 03:07:41.377774 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:07:41.377736    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c005f4a_5b60_45c8_9c90_9a2b89eebc71.slice/crio-99c486cbf0471686d0f2936ffce8e9fbe4fc50dd4f04a153bcea86a00c088b93 WatchSource:0}: Error finding container 99c486cbf0471686d0f2936ffce8e9fbe4fc50dd4f04a153bcea86a00c088b93: Status 404 returned error can't find the container with id 99c486cbf0471686d0f2936ffce8e9fbe4fc50dd4f04a153bcea86a00c088b93
Oct 31 03:07:41.720299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:41.720211    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c979ccc9-r9gs5" event={"ID":"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684","Type":"ContainerStarted","Data":"ab627dce0a94d6113cb8a0fc7e467464cf008fa9e813e5ced4c0f8985df58b87"}
Oct 31 03:07:41.722515 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:41.722480    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerStarted","Data":"99c486cbf0471686d0f2936ffce8e9fbe4fc50dd4f04a153bcea86a00c088b93"}
Oct 31 03:07:41.844585 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:41.844552    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2bqmj"
Oct 31 03:07:42.419763 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:42.419331    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:42.425813 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:42.425783    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:42.734981 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:42.734936    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj" event={"ID":"a18739ba-0914-4b52-bcda-4a5fafe22225","Type":"ContainerStarted","Data":"4f6418aeba839783c8843671ab7ed352d1f6c07d29666ef39ea655c7e5b1e807"}
Oct 31 03:07:42.737408 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:42.737369    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6" event={"ID":"a4d955b0-1418-499f-9036-961443f1e130","Type":"ContainerStarted","Data":"1774d39a68fc917d8dda9200ce04aee622ce8a84ef9f41677076263b7e7a7106"}
Oct 31 03:07:43.743371 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:43.743332    5415 generic.go:356] "Generic (PLEG): container finished" podID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerID="09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb" exitCode=0
Oct 31 03:07:43.743872 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:43.743461    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerDied","Data":"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"}
Oct 31 03:07:43.743872 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:43.743517    5415 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:07:43.811041 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:43.811004    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:45.806934 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:45.806463    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:07:46.184185 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:07:46.184143    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:07:46.389027 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:46.388956    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtgvk"
Oct 31 03:07:46.576383 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:46.576307    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:47.540750 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:47.540705    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:47.541362 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:47.540895    5415 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:07:47.546346 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:47.546271    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:07:48.315459 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:48.315379    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:07:48.315635 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:48.315619    5415 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:07:48.766103 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:48.766060    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerStarted","Data":"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"}
Oct 31 03:07:48.766549 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:48.766146    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerStarted","Data":"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"}
Oct 31 03:07:49.523289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:49.523249    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:49.523777 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:49.523756    5415 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:07:49.529737 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:49.529706    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p2kxj"
Oct 31 03:07:51.785357 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:51.785267    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerStarted","Data":"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"}
Oct 31 03:07:51.785796 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:51.785359    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerStarted","Data":"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"}
Oct 31 03:07:51.785796 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:51.785403    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerStarted","Data":"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"}
Oct 31 03:07:51.785796 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:51.785442    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerStarted","Data":"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"}
Oct 31 03:07:52.045279 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:52.045155    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.035789524 podStartE2EDuration="12.045134617s" totalImagesPullingTime="8.986932449s" totalInitContainerRuntime="22.412644ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:51.997705237 +0000 UTC m=+13.558184986" watchObservedRunningTime="2026-10-31 03:07:52.045134617 +0000 UTC m=+13.605614351"
Oct 31 03:07:53.083402 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:53.083355    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:07:53.910847 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:53.910809    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:07:53.918187 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:53.918159    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x4wzm"
Oct 31 03:07:56.479910 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:56.479753    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c979ccc9-r9gs5"]
Oct 31 03:07:59.834346 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:59.834297    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7rr4c" event={"ID":"6cb392b3-c5a6-4ae1-af06-82f167ed3c27","Type":"ContainerStarted","Data":"db98bfe6fa460dfdebea61c0d34dace7bec1cb3700a4cdefe8272f8dd04b3379"}
Oct 31 03:07:59.834940 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:59.834649    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:07:59.850832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:07:59.850800    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-7rr4c"
Oct 31 03:08:03.817660 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:03.817620    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:08:03.822138 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:03.822106    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7887dcbc55-2plp6"
Oct 31 03:08:05.811685 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:05.811651    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mr7hm"
Oct 31 03:08:06.566842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:06.566813    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7ws"
Oct 31 03:08:16.098929 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:16.098869    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:16.140696 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:16.140662    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:16.173929 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:08:16.173870    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:08:16.915596 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:16.915569    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:21.518829 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.518786    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-74c979ccc9-r9gs5" podUID="b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" containerName="console" containerID="cri-o://ab627dce0a94d6113cb8a0fc7e467464cf008fa9e813e5ced4c0f8985df58b87" gracePeriod=15
Oct 31 03:08:21.810121 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.810087    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:08:21.887623 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.887581    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-tmp\") pod \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") "
Oct 31 03:08:21.887856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.887639    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-oauth-config\") pod \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") "
Oct 31 03:08:21.887856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.887673    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-config\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-config\") pod \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") "
Oct 31 03:08:21.887856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.887712    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-oauth-serving-cert\") pod \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") "
Oct 31 03:08:21.887856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.887753    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-service-ca\") pod \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") "
Oct 31 03:08:21.887856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.887809    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-serving-cert\") pod \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") "
Oct 31 03:08:21.888165 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.887911    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-kube-api-access-6zvxt\" (UniqueName: \"kubernetes.io/projected/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-kube-api-access-6zvxt\") pod \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\" (UID: \"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684\") "
Oct 31 03:08:21.888165 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.887954    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-tmp" pod "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" (UID: "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:08:21.888165 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.888151    5415 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-tmp\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:21.888361 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.888258    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-config" pod "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" (UID: "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:21.888361 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.888274    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-oauth-serving-cert" pod "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" (UID: "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:21.888361 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.888263    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-service-ca" pod "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" (UID: "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:21.890304 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.890275    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-serving-cert" pod "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" (UID: "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:21.890413 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.890326    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-kube-api-access-6zvxt" pod "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" (UID: "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684"). InnerVolumeSpecName "kube-api-access-6zvxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:08:21.890413 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.890331    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-oauth-config" pod "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" (UID: "b5b43c0c-e9fc-4d01-8034-2c1d10c3c684"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:21.918403 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.918367    5415 generic.go:356] "Generic (PLEG): container finished" podID="b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" containerID="ab627dce0a94d6113cb8a0fc7e467464cf008fa9e813e5ced4c0f8985df58b87" exitCode=0
Oct 31 03:08:21.918587 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.918416    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c979ccc9-r9gs5"
Oct 31 03:08:21.918587 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.918445    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c979ccc9-r9gs5" event={"ID":"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684","Type":"ContainerDied","Data":"ab627dce0a94d6113cb8a0fc7e467464cf008fa9e813e5ced4c0f8985df58b87"}
Oct 31 03:08:21.918587 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.918499    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c979ccc9-r9gs5" event={"ID":"b5b43c0c-e9fc-4d01-8034-2c1d10c3c684","Type":"ContainerDied","Data":"5a19334855370632f158cab70d5975524cf2ce6394dcc4c55839ea58a91013d4"}
Oct 31 03:08:21.918587 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.918520    5415 scope.go:122] "RemoveContainer" containerID="ab627dce0a94d6113cb8a0fc7e467464cf008fa9e813e5ced4c0f8985df58b87"
Oct 31 03:08:21.930183 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.930159    5415 scope.go:122] "RemoveContainer" containerID="ab627dce0a94d6113cb8a0fc7e467464cf008fa9e813e5ced4c0f8985df58b87"
Oct 31 03:08:21.930568 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:08:21.930546    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab627dce0a94d6113cb8a0fc7e467464cf008fa9e813e5ced4c0f8985df58b87\": container with ID starting with ab627dce0a94d6113cb8a0fc7e467464cf008fa9e813e5ced4c0f8985df58b87 not found: ID does not exist" containerID="ab627dce0a94d6113cb8a0fc7e467464cf008fa9e813e5ced4c0f8985df58b87"
Oct 31 03:08:21.930662 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.930576    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab627dce0a94d6113cb8a0fc7e467464cf008fa9e813e5ced4c0f8985df58b87"} err="failed to get container status \"ab627dce0a94d6113cb8a0fc7e467464cf008fa9e813e5ced4c0f8985df58b87\": rpc error: code = NotFound desc = could not find container \"ab627dce0a94d6113cb8a0fc7e467464cf008fa9e813e5ced4c0f8985df58b87\": container with ID starting with ab627dce0a94d6113cb8a0fc7e467464cf008fa9e813e5ced4c0f8985df58b87 not found: ID does not exist"
Oct 31 03:08:21.989314 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.989278    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zvxt\" (UniqueName: \"kubernetes.io/projected/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-kube-api-access-6zvxt\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:21.989314 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.989307    5415 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-oauth-config\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:21.989314 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.989319    5415 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-config\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:21.989566 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.989329    5415 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-oauth-serving-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:21.989566 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.989338    5415 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:21.989566 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:21.989347    5415 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684-console-serving-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:22.073080 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:22.072993    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c979ccc9-r9gs5"]
Oct 31 03:08:22.077173 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:22.077145    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74c979ccc9-r9gs5"]
Oct 31 03:08:22.603728 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:22.603687    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" path="/var/lib/kubelet/pods/b5b43c0c-e9fc-4d01-8034-2c1d10c3c684/volumes"
Oct 31 03:08:33.904895 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:33.904847    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 03:08:33.905349 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:33.905291    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="kube-rbac-proxy" containerID="cri-o://464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4" gracePeriod=600
Oct 31 03:08:33.905435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:33.905325    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="kube-rbac-proxy-thanos" containerID="cri-o://62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19" gracePeriod=600
Oct 31 03:08:33.905435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:33.905339    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="thanos-sidecar" containerID="cri-o://6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1" gracePeriod=600
Oct 31 03:08:33.905435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:33.905391    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="config-reloader" containerID="cri-o://9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476" gracePeriod=600
Oct 31 03:08:33.905435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:33.905339    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="kube-rbac-proxy-web" containerID="cri-o://9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb" gracePeriod=600
Oct 31 03:08:33.905630 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:33.905272    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="prometheus" containerID="cri-o://a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657" gracePeriod=600
Oct 31 03:08:34.205754 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.205722    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:34.297800 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.297761    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.297800 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.297803    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-kube-rbac-proxy\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298087 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.297828    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-thanos-prometheus-http-client-file\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298087 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.297870    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298087 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298044    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-1\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298087 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298079    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-serving-certs-ca-bundle\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298305 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298103    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-kube-api-access-zjhm9\" (UniqueName: \"kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-kube-api-access-zjhm9\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298305 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298139    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-grpc-tls\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298305 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298181    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-2\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298305 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298222    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-tls-assets\" (UniqueName: \"kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-tls-assets\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298305 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298250    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-db\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298305 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298284    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298314    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-tls\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298358    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-metrics-client-ca\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298397    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-kubelet-serving-ca-bundle\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298436    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-trusted-ca-bundle\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298493    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-0\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298525    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-metrics-client-certs\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298561    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config-out\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.298594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298595    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-web-config\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-web-config\") pod \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\" (UID: \"8c005f4a-5b60-45c8-9c90-9a2b89eebc71\") "
Oct 31 03:08:34.299548 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.298430    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-1" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:34.300207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.299197    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-2" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:34.300207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.299392    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-db" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:08:34.300207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.299681    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-serving-certs-ca-bundle" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:34.300207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.299928    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-kubelet-serving-ca-bundle" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:34.300207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.299937    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-metrics-client-ca" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:34.300207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.300124    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-trusted-ca-bundle" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:34.300745 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.300688    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-thanos-prometheus-http-client-file" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:34.301153 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.301107    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-0" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:34.301371 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.301338    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-kube-rbac-proxy-web" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:34.301587 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.301557    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:34.302166 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.302135    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-thanos-sidecar-tls" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:34.302397 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.302363    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-kube-rbac-proxy" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:34.302686 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.302642    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config-out" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:08:34.302783 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.302765    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-metrics-client-certs" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:34.303145 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.303128    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-kube-api-access-zjhm9" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "kube-api-access-zjhm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:08:34.303778 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.303756    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-grpc-tls" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:34.303853 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.303771    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-tls" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:34.303853 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.303799    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-tls-assets" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:08:34.312910 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.312851    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-web-config" pod "8c005f4a-5b60-45c8-9c90-9a2b89eebc71" (UID: "8c005f4a-5b60-45c8-9c90-9a2b89eebc71"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:34.400248 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400214    5415 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-web-config\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400248 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400243    5415 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400248 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400254    5415 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-kube-rbac-proxy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400271    5415 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400281    5415 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400289    5415 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400298    5415 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400311    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjhm9\" (UniqueName: \"kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-kube-api-access-zjhm9\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400319    5415 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-grpc-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400327    5415 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400338    5415 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-tls-assets\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400351    5415 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-db\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400360    5415 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400369    5415 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400377    5415 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-metrics-client-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400386    5415 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400394    5415 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400404    5415 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400415    5415 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-secret-metrics-client-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.400490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.400427    5415 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c005f4a-5b60-45c8-9c90-9a2b89eebc71-config-out\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:34.974290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974258    5415 generic.go:356] "Generic (PLEG): container finished" podID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerID="62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19" exitCode=0
Oct 31 03:08:34.974290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974282    5415 generic.go:356] "Generic (PLEG): container finished" podID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerID="464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4" exitCode=0
Oct 31 03:08:34.974290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974288    5415 generic.go:356] "Generic (PLEG): container finished" podID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerID="9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb" exitCode=0
Oct 31 03:08:34.974290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974294    5415 generic.go:356] "Generic (PLEG): container finished" podID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerID="6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1" exitCode=0
Oct 31 03:08:34.974832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974303    5415 generic.go:356] "Generic (PLEG): container finished" podID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerID="9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476" exitCode=0
Oct 31 03:08:34.974832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974308    5415 generic.go:356] "Generic (PLEG): container finished" podID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerID="a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657" exitCode=0
Oct 31 03:08:34.974832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974340    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:34.974832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974336    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerDied","Data":"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"}
Oct 31 03:08:34.974832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974479    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerDied","Data":"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"}
Oct 31 03:08:34.974832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974506    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerDied","Data":"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"}
Oct 31 03:08:34.974832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974531    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerDied","Data":"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"}
Oct 31 03:08:34.974832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974542    5415 scope.go:122] "RemoveContainer" containerID="62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"
Oct 31 03:08:34.974832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974556    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerDied","Data":"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"}
Oct 31 03:08:34.974832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974583    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerDied","Data":"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"}
Oct 31 03:08:34.974832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.974606    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c005f4a-5b60-45c8-9c90-9a2b89eebc71","Type":"ContainerDied","Data":"99c486cbf0471686d0f2936ffce8e9fbe4fc50dd4f04a153bcea86a00c088b93"}
Oct 31 03:08:34.982190 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.982170    5415 scope.go:122] "RemoveContainer" containerID="464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"
Oct 31 03:08:34.989417 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.989396    5415 scope.go:122] "RemoveContainer" containerID="9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"
Oct 31 03:08:34.996508 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:34.996487    5415 scope.go:122] "RemoveContainer" containerID="6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"
Oct 31 03:08:35.003358 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.003338    5415 scope.go:122] "RemoveContainer" containerID="9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"
Oct 31 03:08:35.010431 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.010409    5415 scope.go:122] "RemoveContainer" containerID="a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"
Oct 31 03:08:35.017897 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.017861    5415 scope.go:122] "RemoveContainer" containerID="09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"
Oct 31 03:08:35.024676 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.024656    5415 scope.go:122] "RemoveContainer" containerID="62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"
Oct 31 03:08:35.024990 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:08:35.024966    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19\": container with ID starting with 62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19 not found: ID does not exist" containerID="62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"
Oct 31 03:08:35.025058 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.025006    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"} err="failed to get container status \"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19\": rpc error: code = NotFound desc = could not find container \"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19\": container with ID starting with 62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19 not found: ID does not exist"
Oct 31 03:08:35.025058 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.025033    5415 scope.go:122] "RemoveContainer" containerID="464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"
Oct 31 03:08:35.025321 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:08:35.025289    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4\": container with ID starting with 464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4 not found: ID does not exist" containerID="464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"
Oct 31 03:08:35.025360 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.025327    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"} err="failed to get container status \"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4\": rpc error: code = NotFound desc = could not find container \"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4\": container with ID starting with 464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4 not found: ID does not exist"
Oct 31 03:08:35.025360 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.025342    5415 scope.go:122] "RemoveContainer" containerID="9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"
Oct 31 03:08:35.025588 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:08:35.025573    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb\": container with ID starting with 9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb not found: ID does not exist" containerID="9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"
Oct 31 03:08:35.025647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.025592    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"} err="failed to get container status \"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb\": rpc error: code = NotFound desc = could not find container \"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb\": container with ID starting with 9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb not found: ID does not exist"
Oct 31 03:08:35.025647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.025610    5415 scope.go:122] "RemoveContainer" containerID="6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"
Oct 31 03:08:35.025905 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:08:35.025865    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1\": container with ID starting with 6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1 not found: ID does not exist" containerID="6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"
Oct 31 03:08:35.025962 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.025916    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"} err="failed to get container status \"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1\": rpc error: code = NotFound desc = could not find container \"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1\": container with ID starting with 6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1 not found: ID does not exist"
Oct 31 03:08:35.025962 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.025937    5415 scope.go:122] "RemoveContainer" containerID="9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"
Oct 31 03:08:35.026208 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:08:35.026188    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476\": container with ID starting with 9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476 not found: ID does not exist" containerID="9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"
Oct 31 03:08:35.026301 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.026211    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"} err="failed to get container status \"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476\": rpc error: code = NotFound desc = could not find container \"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476\": container with ID starting with 9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476 not found: ID does not exist"
Oct 31 03:08:35.026301 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.026227    5415 scope.go:122] "RemoveContainer" containerID="a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"
Oct 31 03:08:35.026485 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:08:35.026472    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657\": container with ID starting with a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657 not found: ID does not exist" containerID="a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"
Oct 31 03:08:35.026530 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.026488    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"} err="failed to get container status \"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657\": rpc error: code = NotFound desc = could not find container \"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657\": container with ID starting with a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657 not found: ID does not exist"
Oct 31 03:08:35.026530 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.026501    5415 scope.go:122] "RemoveContainer" containerID="09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"
Oct 31 03:08:35.026697 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:08:35.026681    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb\": container with ID starting with 09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb not found: ID does not exist" containerID="09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"
Oct 31 03:08:35.026731 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.026702    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"} err="failed to get container status \"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb\": rpc error: code = NotFound desc = could not find container \"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb\": container with ID starting with 09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb not found: ID does not exist"
Oct 31 03:08:35.026731 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.026715    5415 scope.go:122] "RemoveContainer" containerID="62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"
Oct 31 03:08:35.026968 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.026948    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"} err="failed to get container status \"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19\": rpc error: code = NotFound desc = could not find container \"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19\": container with ID starting with 62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19 not found: ID does not exist"
Oct 31 03:08:35.027016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.026969    5415 scope.go:122] "RemoveContainer" containerID="464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"
Oct 31 03:08:35.027167 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.027152    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"} err="failed to get container status \"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4\": rpc error: code = NotFound desc = could not find container \"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4\": container with ID starting with 464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4 not found: ID does not exist"
Oct 31 03:08:35.027206 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.027167    5415 scope.go:122] "RemoveContainer" containerID="9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"
Oct 31 03:08:35.027360 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.027345    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"} err="failed to get container status \"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb\": rpc error: code = NotFound desc = could not find container \"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb\": container with ID starting with 9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb not found: ID does not exist"
Oct 31 03:08:35.027408 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.027360    5415 scope.go:122] "RemoveContainer" containerID="6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"
Oct 31 03:08:35.027542 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.027527    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"} err="failed to get container status \"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1\": rpc error: code = NotFound desc = could not find container \"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1\": container with ID starting with 6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1 not found: ID does not exist"
Oct 31 03:08:35.027582 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.027542    5415 scope.go:122] "RemoveContainer" containerID="9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"
Oct 31 03:08:35.027749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.027728    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"} err="failed to get container status \"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476\": rpc error: code = NotFound desc = could not find container \"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476\": container with ID starting with 9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476 not found: ID does not exist"
Oct 31 03:08:35.027749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.027747    5415 scope.go:122] "RemoveContainer" containerID="a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"
Oct 31 03:08:35.028016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.028001    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"} err="failed to get container status \"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657\": rpc error: code = NotFound desc = could not find container \"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657\": container with ID starting with a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657 not found: ID does not exist"
Oct 31 03:08:35.028016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.028015    5415 scope.go:122] "RemoveContainer" containerID="09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"
Oct 31 03:08:35.028234 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.028215    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"} err="failed to get container status \"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb\": rpc error: code = NotFound desc = could not find container \"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb\": container with ID starting with 09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb not found: ID does not exist"
Oct 31 03:08:35.028298 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.028233    5415 scope.go:122] "RemoveContainer" containerID="62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"
Oct 31 03:08:35.028455 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.028439    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"} err="failed to get container status \"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19\": rpc error: code = NotFound desc = could not find container \"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19\": container with ID starting with 62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19 not found: ID does not exist"
Oct 31 03:08:35.028455 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.028454    5415 scope.go:122] "RemoveContainer" containerID="464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"
Oct 31 03:08:35.028634 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.028616    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"} err="failed to get container status \"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4\": rpc error: code = NotFound desc = could not find container \"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4\": container with ID starting with 464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4 not found: ID does not exist"
Oct 31 03:08:35.028676 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.028634    5415 scope.go:122] "RemoveContainer" containerID="9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"
Oct 31 03:08:35.028816 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.028795    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"} err="failed to get container status \"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb\": rpc error: code = NotFound desc = could not find container \"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb\": container with ID starting with 9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb not found: ID does not exist"
Oct 31 03:08:35.028816 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.028812    5415 scope.go:122] "RemoveContainer" containerID="6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"
Oct 31 03:08:35.029032 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.029015    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"} err="failed to get container status \"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1\": rpc error: code = NotFound desc = could not find container \"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1\": container with ID starting with 6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1 not found: ID does not exist"
Oct 31 03:08:35.029070 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.029031    5415 scope.go:122] "RemoveContainer" containerID="9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"
Oct 31 03:08:35.029244 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.029230    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"} err="failed to get container status \"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476\": rpc error: code = NotFound desc = could not find container \"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476\": container with ID starting with 9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476 not found: ID does not exist"
Oct 31 03:08:35.029279 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.029244    5415 scope.go:122] "RemoveContainer" containerID="a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"
Oct 31 03:08:35.029445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.029424    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"} err="failed to get container status \"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657\": rpc error: code = NotFound desc = could not find container \"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657\": container with ID starting with a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657 not found: ID does not exist"
Oct 31 03:08:35.029494 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.029445    5415 scope.go:122] "RemoveContainer" containerID="09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"
Oct 31 03:08:35.029677 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.029659    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"} err="failed to get container status \"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb\": rpc error: code = NotFound desc = could not find container \"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb\": container with ID starting with 09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb not found: ID does not exist"
Oct 31 03:08:35.029722 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.029677    5415 scope.go:122] "RemoveContainer" containerID="62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"
Oct 31 03:08:35.029858 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.029842    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"} err="failed to get container status \"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19\": rpc error: code = NotFound desc = could not find container \"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19\": container with ID starting with 62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19 not found: ID does not exist"
Oct 31 03:08:35.029933 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.029857    5415 scope.go:122] "RemoveContainer" containerID="464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"
Oct 31 03:08:35.030068 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.030053    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"} err="failed to get container status \"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4\": rpc error: code = NotFound desc = could not find container \"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4\": container with ID starting with 464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4 not found: ID does not exist"
Oct 31 03:08:35.030106 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.030067    5415 scope.go:122] "RemoveContainer" containerID="9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"
Oct 31 03:08:35.030265 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.030250    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"} err="failed to get container status \"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb\": rpc error: code = NotFound desc = could not find container \"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb\": container with ID starting with 9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb not found: ID does not exist"
Oct 31 03:08:35.030319 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.030265    5415 scope.go:122] "RemoveContainer" containerID="6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"
Oct 31 03:08:35.030463 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.030436    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"} err="failed to get container status \"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1\": rpc error: code = NotFound desc = could not find container \"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1\": container with ID starting with 6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1 not found: ID does not exist"
Oct 31 03:08:35.030504 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.030463    5415 scope.go:122] "RemoveContainer" containerID="9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"
Oct 31 03:08:35.030652 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.030632    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"} err="failed to get container status \"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476\": rpc error: code = NotFound desc = could not find container \"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476\": container with ID starting with 9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476 not found: ID does not exist"
Oct 31 03:08:35.030723 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.030654    5415 scope.go:122] "RemoveContainer" containerID="a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"
Oct 31 03:08:35.030868 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.030854    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"} err="failed to get container status \"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657\": rpc error: code = NotFound desc = could not find container \"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657\": container with ID starting with a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657 not found: ID does not exist"
Oct 31 03:08:35.030936 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.030867    5415 scope.go:122] "RemoveContainer" containerID="09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"
Oct 31 03:08:35.031084 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.031062    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"} err="failed to get container status \"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb\": rpc error: code = NotFound desc = could not find container \"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb\": container with ID starting with 09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb not found: ID does not exist"
Oct 31 03:08:35.031125 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.031085    5415 scope.go:122] "RemoveContainer" containerID="62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"
Oct 31 03:08:35.031269 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.031252    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"} err="failed to get container status \"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19\": rpc error: code = NotFound desc = could not find container \"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19\": container with ID starting with 62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19 not found: ID does not exist"
Oct 31 03:08:35.031309 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.031272    5415 scope.go:122] "RemoveContainer" containerID="464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"
Oct 31 03:08:35.031462 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.031442    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"} err="failed to get container status \"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4\": rpc error: code = NotFound desc = could not find container \"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4\": container with ID starting with 464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4 not found: ID does not exist"
Oct 31 03:08:35.031535 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.031464    5415 scope.go:122] "RemoveContainer" containerID="9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"
Oct 31 03:08:35.031655 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.031638    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"} err="failed to get container status \"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb\": rpc error: code = NotFound desc = could not find container \"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb\": container with ID starting with 9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb not found: ID does not exist"
Oct 31 03:08:35.031655 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.031653    5415 scope.go:122] "RemoveContainer" containerID="6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"
Oct 31 03:08:35.031839 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.031825    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"} err="failed to get container status \"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1\": rpc error: code = NotFound desc = could not find container \"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1\": container with ID starting with 6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1 not found: ID does not exist"
Oct 31 03:08:35.031932 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.031838    5415 scope.go:122] "RemoveContainer" containerID="9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"
Oct 31 03:08:35.032062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.032043    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"} err="failed to get container status \"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476\": rpc error: code = NotFound desc = could not find container \"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476\": container with ID starting with 9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476 not found: ID does not exist"
Oct 31 03:08:35.032102 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.032071    5415 scope.go:122] "RemoveContainer" containerID="a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"
Oct 31 03:08:35.032275 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.032260    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"} err="failed to get container status \"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657\": rpc error: code = NotFound desc = could not find container \"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657\": container with ID starting with a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657 not found: ID does not exist"
Oct 31 03:08:35.032318 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.032275    5415 scope.go:122] "RemoveContainer" containerID="09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"
Oct 31 03:08:35.032468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.032451    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"} err="failed to get container status \"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb\": rpc error: code = NotFound desc = could not find container \"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb\": container with ID starting with 09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb not found: ID does not exist"
Oct 31 03:08:35.032507 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.032468    5415 scope.go:122] "RemoveContainer" containerID="62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"
Oct 31 03:08:35.032652 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.032636    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19"} err="failed to get container status \"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19\": rpc error: code = NotFound desc = could not find container \"62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19\": container with ID starting with 62e8ee592a94db012a66da9c0f26ba454d686f4183aa788f14086940ab2ecd19 not found: ID does not exist"
Oct 31 03:08:35.032697 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.032652    5415 scope.go:122] "RemoveContainer" containerID="464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"
Oct 31 03:08:35.032842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.032826    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4"} err="failed to get container status \"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4\": rpc error: code = NotFound desc = could not find container \"464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4\": container with ID starting with 464e3570337679e19722ed56e1dc7d35e9f6cb25fe5bd575f9e92647f7ca9ad4 not found: ID does not exist"
Oct 31 03:08:35.032901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.032841    5415 scope.go:122] "RemoveContainer" containerID="9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"
Oct 31 03:08:35.033036 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.033021    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb"} err="failed to get container status \"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb\": rpc error: code = NotFound desc = could not find container \"9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb\": container with ID starting with 9d1f33010a439b9589b6bf6ad7c0a4d30019bd323e99ed0da765df24a2827feb not found: ID does not exist"
Oct 31 03:08:35.033079 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.033036    5415 scope.go:122] "RemoveContainer" containerID="6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"
Oct 31 03:08:35.033253 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.033237    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1"} err="failed to get container status \"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1\": rpc error: code = NotFound desc = could not find container \"6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1\": container with ID starting with 6c69f8ff0b99dbaacf6d620a5c4d6d79c7a4e99a634b8c2206d729b84881b3d1 not found: ID does not exist"
Oct 31 03:08:35.033295 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.033253    5415 scope.go:122] "RemoveContainer" containerID="9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"
Oct 31 03:08:35.033429 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.033413    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476"} err="failed to get container status \"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476\": rpc error: code = NotFound desc = could not find container \"9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476\": container with ID starting with 9b03a90e022d3afac88b4f75d71aecf77f4ea18e7ff51c12552b4ec1a28c0476 not found: ID does not exist"
Oct 31 03:08:35.033429 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.033428    5415 scope.go:122] "RemoveContainer" containerID="a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"
Oct 31 03:08:35.033695 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.033655    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657"} err="failed to get container status \"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657\": rpc error: code = NotFound desc = could not find container \"a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657\": container with ID starting with a94722da6c1c41da956ce6f60976afe7466378bb944a175a0333086fb0ea8657 not found: ID does not exist"
Oct 31 03:08:35.033737 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.033696    5415 scope.go:122] "RemoveContainer" containerID="09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"
Oct 31 03:08:35.033918 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.033896    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb"} err="failed to get container status \"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb\": rpc error: code = NotFound desc = could not find container \"09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb\": container with ID starting with 09c5af7041415cdda0e13f10843ab7734b58ff57d7e45d6298e797ea95da14fb not found: ID does not exist"
Oct 31 03:08:35.134499 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.134464    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.183097    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184156    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184704    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="config-reloader"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184721    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="config-reloader"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184737    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="thanos-sidecar"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184745    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="thanos-sidecar"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184762    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="kube-rbac-proxy"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184770    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="kube-rbac-proxy"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184805    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="kube-rbac-proxy-web"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184813    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="kube-rbac-proxy-web"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184827    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="init-config-reloader"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184834    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="init-config-reloader"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184852    5415 cpu_manager.go:414] "containerMap: removing container" podUID="b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" containerName="console"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184859    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" containerName="console"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184869    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="prometheus"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184892    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="prometheus"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184920    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="kube-rbac-proxy-thanos"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.184928    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="kube-rbac-proxy-thanos"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.185050    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="kube-rbac-proxy-web"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.185069    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b43c0c-e9fc-4d01-8034-2c1d10c3c684" containerName="console"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.185077    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="prometheus"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.185095    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="kube-rbac-proxy-thanos"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.185104    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="config-reloader"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.185121    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="thanos-sidecar"
Oct 31 03:08:35.185337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.185128    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" containerName="kube-rbac-proxy"
Oct 31 03:08:35.222790 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.222708    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.273529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.273492    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 03:08:35.273712 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.273562    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 03:08:35.273712 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.273567    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 03:08:35.273712 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.273635    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 03:08:35.273712 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.273656    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 03:08:35.273712 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.273657    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 03:08:35.274016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.273713    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 03:08:35.274016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.273739    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 03:08:35.274016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.273966    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 03:08:35.274139 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.274069    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 03:08:35.274258 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.274243    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k8rng\""
Oct 31 03:08:35.274321 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.274299    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5vkgn06l0rnmt\""
Oct 31 03:08:35.274377 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.274362    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 03:08:35.274429 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.274417    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 03:08:35.306893 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.306853    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72e90f94-e4f3-4fd3-a667-de34caeb2d48-config-out\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307055 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.306904    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307055 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.306974    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72e90f94-e4f3-4fd3-a667-de34caeb2d48-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307055 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.306998    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307055 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307032    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307187 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307092    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307187 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307114    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-config\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307187 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307135    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307187 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307151    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307331 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307213    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307331 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307246    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307331 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307278    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307331 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307297    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307331 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307313    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307553 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307419    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307553 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307447    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307553 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307469    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-web-config\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307553 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307486    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307704 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307562    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.307704 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.307620    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pxnv\" (UniqueName: \"kubernetes.io/projected/72e90f94-e4f3-4fd3-a667-de34caeb2d48-kube-api-access-8pxnv\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.321550 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.321527    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 03:08:35.321684 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.321564    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 03:08:35.341744 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.341708    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 03:08:35.408220 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408186    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.408220 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408221    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.408491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408243    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-config\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.408491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408266    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.408491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408282    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.408491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408303    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.408491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408322    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.408491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408347    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.408491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408366    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.408491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408389    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.408491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408421    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.408491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408445    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.408491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408467    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-web-config\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.408491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408494    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.409095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408526    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.409095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408553    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pxnv\" (UniqueName: \"kubernetes.io/projected/72e90f94-e4f3-4fd3-a667-de34caeb2d48-kube-api-access-8pxnv\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.409095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408627    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72e90f94-e4f3-4fd3-a667-de34caeb2d48-config-out\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.409095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408653    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.409095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408678    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72e90f94-e4f3-4fd3-a667-de34caeb2d48-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.409095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408704    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.409095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.408861    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.409493 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.409113    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.409493 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.409392    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.410490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.410181    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.410490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.410223    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.411516 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.411361    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.411516 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.411454    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.412019 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.411989    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.412115 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.412014    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.412297 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.412274    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-config\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.412383 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.412367    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.412645 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.412578    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.412739 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.412655    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.413809 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.413784    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.414006 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.413981    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72e90f94-e4f3-4fd3-a667-de34caeb2d48-config-out\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.414535 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.414514    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72e90f94-e4f3-4fd3-a667-de34caeb2d48-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.414635 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.414614    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.414687 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.414638    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72e90f94-e4f3-4fd3-a667-de34caeb2d48-web-config\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.414871 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.414846    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/72e90f94-e4f3-4fd3-a667-de34caeb2d48-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.464068 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.464032    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pxnv\" (UniqueName: \"kubernetes.io/projected/72e90f94-e4f3-4fd3-a667-de34caeb2d48-kube-api-access-8pxnv\") pod \"prometheus-k8s-0\" (UID: \"72e90f94-e4f3-4fd3-a667-de34caeb2d48\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:35.629080 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:35.629032    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:36.128085 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:36.128052    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 03:08:36.131183 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:08:36.131143    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e90f94_e4f3_4fd3_a667_de34caeb2d48.slice/crio-32a0281c7d9ef7cebfede8c337be7b3031b0613863bcbbbdea7956420004f820 WatchSource:0}: Error finding container 32a0281c7d9ef7cebfede8c337be7b3031b0613863bcbbbdea7956420004f820: Status 404 returned error can't find the container with id 32a0281c7d9ef7cebfede8c337be7b3031b0613863bcbbbdea7956420004f820
Oct 31 03:08:36.603035 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:36.603003    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c005f4a-5b60-45c8-9c90-9a2b89eebc71" path="/var/lib/kubelet/pods/8c005f4a-5b60-45c8-9c90-9a2b89eebc71/volumes"
Oct 31 03:08:36.986314 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:36.986279    5415 generic.go:356] "Generic (PLEG): container finished" podID="72e90f94-e4f3-4fd3-a667-de34caeb2d48" containerID="8c057af4a8fcbc111fed8b7c121a2c5511a74b8e73b9d22c551ec0e984a0b842" exitCode=0
Oct 31 03:08:36.986513 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:36.986368    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72e90f94-e4f3-4fd3-a667-de34caeb2d48","Type":"ContainerDied","Data":"8c057af4a8fcbc111fed8b7c121a2c5511a74b8e73b9d22c551ec0e984a0b842"}
Oct 31 03:08:36.986513 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:36.986455    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72e90f94-e4f3-4fd3-a667-de34caeb2d48","Type":"ContainerStarted","Data":"32a0281c7d9ef7cebfede8c337be7b3031b0613863bcbbbdea7956420004f820"}
Oct 31 03:08:37.994413 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:37.994373    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72e90f94-e4f3-4fd3-a667-de34caeb2d48","Type":"ContainerStarted","Data":"0c95c66bb880e2fe21c5441fd999385a686f75e008d40bd701fa0dfe732a8df6"}
Oct 31 03:08:37.994870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:37.994491    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72e90f94-e4f3-4fd3-a667-de34caeb2d48","Type":"ContainerStarted","Data":"9f2fb5ff1d6e7898c6d683d9836dd85cfab0f690dafa2fc9454a7265eaf9d6fc"}
Oct 31 03:08:37.994870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:37.994534    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72e90f94-e4f3-4fd3-a667-de34caeb2d48","Type":"ContainerStarted","Data":"0a5c9d9c99dd67f90f9aa6df0d6dba38594ada25cab116ad7878e9d7a4cfc02d"}
Oct 31 03:08:37.994870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:37.994576    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72e90f94-e4f3-4fd3-a667-de34caeb2d48","Type":"ContainerStarted","Data":"27b6c96c4d27819f01de39b1c3754329d288d147dd0d2675fb0d741343891999"}
Oct 31 03:08:37.994870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:37.994610    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72e90f94-e4f3-4fd3-a667-de34caeb2d48","Type":"ContainerStarted","Data":"97cc6b3ca8c45764b96ce3c71d030e83d64fdd11a9774150520f424b4ae24bc8"}
Oct 31 03:08:37.994870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:37.994644    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72e90f94-e4f3-4fd3-a667-de34caeb2d48","Type":"ContainerStarted","Data":"7cfb02bd1c428f0883093d64286ad45ffeed2407224323e6e494b64201c56e28"}
Oct 31 03:08:38.117687 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:38.117628    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.103187809 podStartE2EDuration="3.117614265s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.426456ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:38.116627279 +0000 UTC m=+59.677107010" watchObservedRunningTime="2026-10-31 03:08:38.117614265 +0000 UTC m=+59.678094372"
Oct 31 03:08:40.630041 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:40.629984    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:46.174529 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:08:46.174488    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:08:55.506834 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:55.506776    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:55.522926 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:55.522898    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:08:55.645350 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:08:55.645318    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 03:09:16.174625 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:09:16.174526    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:09:46.173959 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:09:46.173919    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:10:16.174197 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:10:16.174160    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:10:46.173089 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:10:46.173004    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:10:59.251119 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.251080    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"]
Oct 31 03:10:59.253799 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.253776    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.298287 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.298257    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"]
Oct 31 03:10:59.339590 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.339556    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 03:10:59.339807 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.339600    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 03:10:59.339807 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.339613    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 03:10:59.339807 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.339616    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 03:10:59.339807 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.339680    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 03:10:59.339807 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.339743    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-92w4t\""
Oct 31 03:10:59.392372 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.392337    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8hnb\" (UniqueName: \"kubernetes.io/projected/145db2a6-08cd-447f-8488-a06f21f1b1db-kube-api-access-b8hnb\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.392372 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.392378    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/145db2a6-08cd-447f-8488-a06f21f1b1db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.392587 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.392408    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/145db2a6-08cd-447f-8488-a06f21f1b1db-tmp\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.392587 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.392510    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/145db2a6-08cd-447f-8488-a06f21f1b1db-manifests\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.392587 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.392538    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/145db2a6-08cd-447f-8488-a06f21f1b1db-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.392587 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.392557    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/145db2a6-08cd-447f-8488-a06f21f1b1db-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.493122 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.493087    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8hnb\" (UniqueName: \"kubernetes.io/projected/145db2a6-08cd-447f-8488-a06f21f1b1db-kube-api-access-b8hnb\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.493335 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.493132    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/145db2a6-08cd-447f-8488-a06f21f1b1db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.493335 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.493165    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/145db2a6-08cd-447f-8488-a06f21f1b1db-tmp\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.493335 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.493207    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/145db2a6-08cd-447f-8488-a06f21f1b1db-manifests\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.493517 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.493356    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/145db2a6-08cd-447f-8488-a06f21f1b1db-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.493517 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.493403    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/145db2a6-08cd-447f-8488-a06f21f1b1db-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.493654 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.493634    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/145db2a6-08cd-447f-8488-a06f21f1b1db-manifests\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.495714 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.495680    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/145db2a6-08cd-447f-8488-a06f21f1b1db-tmp\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.495858 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.495762    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/145db2a6-08cd-447f-8488-a06f21f1b1db-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.495858 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.495824    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/145db2a6-08cd-447f-8488-a06f21f1b1db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.496199 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.496178    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/145db2a6-08cd-447f-8488-a06f21f1b1db-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.550755 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.550372    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8hnb\" (UniqueName: \"kubernetes.io/projected/145db2a6-08cd-447f-8488-a06f21f1b1db-kube-api-access-b8hnb\") pod \"opendatahub-operator-controller-manager-6f949c7964-nnksg\" (UID: \"145db2a6-08cd-447f-8488-a06f21f1b1db\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:10:59.566970 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:10:59.566934    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:11:00.047422 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:00.047396    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"]
Oct 31 03:11:00.050252 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:11:00.050220    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145db2a6_08cd_447f_8488_a06f21f1b1db.slice/crio-e3f159a6a34ed027c0ff25186cb91b71dc0323ebcfcec2d418a6a372daf8b9bc WatchSource:0}: Error finding container e3f159a6a34ed027c0ff25186cb91b71dc0323ebcfcec2d418a6a372daf8b9bc: Status 404 returned error can't find the container with id e3f159a6a34ed027c0ff25186cb91b71dc0323ebcfcec2d418a6a372daf8b9bc
Oct 31 03:11:00.495022 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:00.494984    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg" event={"ID":"145db2a6-08cd-447f-8488-a06f21f1b1db","Type":"ContainerStarted","Data":"e3f159a6a34ed027c0ff25186cb91b71dc0323ebcfcec2d418a6a372daf8b9bc"}
Oct 31 03:11:05.515409 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:05.515369    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg" event={"ID":"145db2a6-08cd-447f-8488-a06f21f1b1db","Type":"ContainerStarted","Data":"26b9329ea86654cfe08110d344057002be38d492f7df15860ac7bb295ab3e44b"}
Oct 31 03:11:06.521125 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:06.521083    5415 generic.go:356] "Generic (PLEG): container finished" podID="145db2a6-08cd-447f-8488-a06f21f1b1db" containerID="26b9329ea86654cfe08110d344057002be38d492f7df15860ac7bb295ab3e44b" exitCode=0
Oct 31 03:11:06.521545 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:06.521137    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg" event={"ID":"145db2a6-08cd-447f-8488-a06f21f1b1db","Type":"ContainerDied","Data":"26b9329ea86654cfe08110d344057002be38d492f7df15860ac7bb295ab3e44b"}
Oct 31 03:11:07.526733 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:07.526699    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg" event={"ID":"145db2a6-08cd-447f-8488-a06f21f1b1db","Type":"ContainerStarted","Data":"00b8894bf52e698d764af0df043493884babc7ee22618d72e934cac2921714f5"}
Oct 31 03:11:07.527173 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:07.526818    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:11:07.661222 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:07.661117    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg" podStartSLOduration=2.036053584 podStartE2EDuration="8.661103515s" totalImagesPullingTime="6.125369509s" totalInitContainerRuntime="499.680422ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:07.660319998 +0000 UTC m=+209.220799758" watchObservedRunningTime="2026-10-31 03:11:07.661103515 +0000 UTC m=+209.221583249"
Oct 31 03:11:13.182621 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.182583    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn"]
Oct 31 03:11:13.187300 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.187275    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn"
Oct 31 03:11:13.284992 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.284960    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:11:13.284992 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.284990    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 03:11:13.285245 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.285230    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-4rjsl\""
Oct 31 03:11:13.285868 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.285848    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn"]
Oct 31 03:11:13.324074 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.324040    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48181687-39a4-4248-9d9a-d7c078fdde06-tmp\") pod \"jobset-operator-674cd59dc5-lzxbn\" (UID: \"48181687-39a4-4248-9d9a-d7c078fdde06\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn"
Oct 31 03:11:13.324257 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.324123    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj6g6\" (UniqueName: \"kubernetes.io/projected/48181687-39a4-4248-9d9a-d7c078fdde06-kube-api-access-nj6g6\") pod \"jobset-operator-674cd59dc5-lzxbn\" (UID: \"48181687-39a4-4248-9d9a-d7c078fdde06\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn"
Oct 31 03:11:13.425422 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.425384    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj6g6\" (UniqueName: \"kubernetes.io/projected/48181687-39a4-4248-9d9a-d7c078fdde06-kube-api-access-nj6g6\") pod \"jobset-operator-674cd59dc5-lzxbn\" (UID: \"48181687-39a4-4248-9d9a-d7c078fdde06\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn"
Oct 31 03:11:13.425620 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.425444    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48181687-39a4-4248-9d9a-d7c078fdde06-tmp\") pod \"jobset-operator-674cd59dc5-lzxbn\" (UID: \"48181687-39a4-4248-9d9a-d7c078fdde06\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn"
Oct 31 03:11:13.425912 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.425872    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48181687-39a4-4248-9d9a-d7c078fdde06-tmp\") pod \"jobset-operator-674cd59dc5-lzxbn\" (UID: \"48181687-39a4-4248-9d9a-d7c078fdde06\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn"
Oct 31 03:11:13.494614 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.494544    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj6g6\" (UniqueName: \"kubernetes.io/projected/48181687-39a4-4248-9d9a-d7c078fdde06-kube-api-access-nj6g6\") pod \"jobset-operator-674cd59dc5-lzxbn\" (UID: \"48181687-39a4-4248-9d9a-d7c078fdde06\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn"
Oct 31 03:11:13.519527 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.519492    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn"
Oct 31 03:11:13.884366 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:13.884333    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn"]
Oct 31 03:11:13.888134 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:11:13.888102    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48181687_39a4_4248_9d9a_d7c078fdde06.slice/crio-62e18f91d663773f8e9da07b7bb228b107f875777fe824445b237b331bc197e2 WatchSource:0}: Error finding container 62e18f91d663773f8e9da07b7bb228b107f875777fe824445b237b331bc197e2: Status 404 returned error can't find the container with id 62e18f91d663773f8e9da07b7bb228b107f875777fe824445b237b331bc197e2
Oct 31 03:11:14.566650 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:14.566615    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn" event={"ID":"48181687-39a4-4248-9d9a-d7c078fdde06","Type":"ContainerStarted","Data":"62e18f91d663773f8e9da07b7bb228b107f875777fe824445b237b331bc197e2"}
Oct 31 03:11:15.181792 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.181756    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7"]
Oct 31 03:11:15.186676 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.186610    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7"
Oct 31 03:11:15.283233 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.283198    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 03:11:15.283233 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.283219    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 03:11:15.283233 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.283230    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-tvzdr\""
Oct 31 03:11:15.284076 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.284056    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7"]
Oct 31 03:11:15.340802 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.340765    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb7497ad-9228-4877-bca1-ab2ee38d5525-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-w4fx7\" (UID: \"cb7497ad-9228-4877-bca1-ab2ee38d5525\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7"
Oct 31 03:11:15.340802 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.340815    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4bw\" (UniqueName: \"kubernetes.io/projected/cb7497ad-9228-4877-bca1-ab2ee38d5525-kube-api-access-fq4bw\") pod \"cert-manager-webhook-5dbbbf75cc-w4fx7\" (UID: \"cb7497ad-9228-4877-bca1-ab2ee38d5525\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7"
Oct 31 03:11:15.441538 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.441446    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb7497ad-9228-4877-bca1-ab2ee38d5525-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-w4fx7\" (UID: \"cb7497ad-9228-4877-bca1-ab2ee38d5525\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7"
Oct 31 03:11:15.441538 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.441484    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4bw\" (UniqueName: \"kubernetes.io/projected/cb7497ad-9228-4877-bca1-ab2ee38d5525-kube-api-access-fq4bw\") pod \"cert-manager-webhook-5dbbbf75cc-w4fx7\" (UID: \"cb7497ad-9228-4877-bca1-ab2ee38d5525\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7"
Oct 31 03:11:15.496378 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.496344    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb7497ad-9228-4877-bca1-ab2ee38d5525-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-w4fx7\" (UID: \"cb7497ad-9228-4877-bca1-ab2ee38d5525\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7"
Oct 31 03:11:15.496522 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.496479    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4bw\" (UniqueName: \"kubernetes.io/projected/cb7497ad-9228-4877-bca1-ab2ee38d5525-kube-api-access-fq4bw\") pod \"cert-manager-webhook-5dbbbf75cc-w4fx7\" (UID: \"cb7497ad-9228-4877-bca1-ab2ee38d5525\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7"
Oct 31 03:11:15.499274 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.499252    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7"
Oct 31 03:11:15.869177 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:15.869144    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7"]
Oct 31 03:11:15.873109 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:11:15.873073    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb7497ad_9228_4877_bca1_ab2ee38d5525.slice/crio-0acdba17cb9a790ca96fd1157ccd5045b2d51278926f217263fd1ce0ccb9880f WatchSource:0}: Error finding container 0acdba17cb9a790ca96fd1157ccd5045b2d51278926f217263fd1ce0ccb9880f: Status 404 returned error can't find the container with id 0acdba17cb9a790ca96fd1157ccd5045b2d51278926f217263fd1ce0ccb9880f
Oct 31 03:11:16.184985 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:11:16.184896    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:11:16.577012 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:16.576902    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7" event={"ID":"cb7497ad-9228-4877-bca1-ab2ee38d5525","Type":"ContainerStarted","Data":"0acdba17cb9a790ca96fd1157ccd5045b2d51278926f217263fd1ce0ccb9880f"}
Oct 31 03:11:18.532046 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:18.531941    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-nnksg"
Oct 31 03:11:18.590468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:18.590418    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn" event={"ID":"48181687-39a4-4248-9d9a-d7c078fdde06","Type":"ContainerStarted","Data":"f053ec147f3e9230469a499d7c27199d1e149af92d3a405c4bf6e92de8ffeb6b"}
Oct 31 03:11:18.879754 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:18.879682    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lzxbn" podStartSLOduration=2.0583872 podStartE2EDuration="5.879662376s" totalImagesPullingTime="3.821275176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:18.879371086 +0000 UTC m=+220.439850819" watchObservedRunningTime="2026-10-31 03:11:18.879662376 +0000 UTC m=+220.440142121"
Oct 31 03:11:20.603245 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:20.603211    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7" event={"ID":"cb7497ad-9228-4877-bca1-ab2ee38d5525","Type":"ContainerStarted","Data":"bfd3dfd0898ce964df07af8aad8ac01325fe5348083c9207d79e219d23cfad92"}
Oct 31 03:11:20.603245 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:20.603256    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7"
Oct 31 03:11:20.709738 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:20.709636    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7" podStartSLOduration=1.125997158 podStartE2EDuration="5.709620185s" totalImagesPullingTime="4.583623027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:20.709125642 +0000 UTC m=+222.269605393" watchObservedRunningTime="2026-10-31 03:11:20.709620185 +0000 UTC m=+222.270099913"
Oct 31 03:11:26.607749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:26.607720    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4fx7"
Oct 31 03:11:30.615860 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:30.615822    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmdgv"]
Oct 31 03:11:30.619430 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:30.619404    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmdgv"
Oct 31 03:11:30.674609 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:30.674572    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bgrm\" (UniqueName: \"kubernetes.io/projected/6e19afe0-f05b-4294-8d01-8d04cbb5b833-kube-api-access-6bgrm\") pod \"limitador-operator-controller-manager-5778bb4449-nmdgv\" (UID: \"6e19afe0-f05b-4294-8d01-8d04cbb5b833\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmdgv"
Oct 31 03:11:30.722615 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:30.722582    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 03:11:30.722615 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:30.722605    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-v6rqb\""
Oct 31 03:11:30.722848 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:30.722760    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 03:11:30.723838 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:30.723819    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmdgv"]
Oct 31 03:11:30.775554 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:30.775517    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bgrm\" (UniqueName: \"kubernetes.io/projected/6e19afe0-f05b-4294-8d01-8d04cbb5b833-kube-api-access-6bgrm\") pod \"limitador-operator-controller-manager-5778bb4449-nmdgv\" (UID: \"6e19afe0-f05b-4294-8d01-8d04cbb5b833\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmdgv"
Oct 31 03:11:30.831056 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:30.831020    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bgrm\" (UniqueName: \"kubernetes.io/projected/6e19afe0-f05b-4294-8d01-8d04cbb5b833-kube-api-access-6bgrm\") pod \"limitador-operator-controller-manager-5778bb4449-nmdgv\" (UID: \"6e19afe0-f05b-4294-8d01-8d04cbb5b833\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmdgv"
Oct 31 03:11:30.932550 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:30.932450    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmdgv"
Oct 31 03:11:31.298537 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:31.298463    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmdgv"]
Oct 31 03:11:31.301928 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:11:31.301870    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e19afe0_f05b_4294_8d01_8d04cbb5b833.slice/crio-b63d3f445c17994573af3ec3b2c856d310183c0b717beb740a95493295435877 WatchSource:0}: Error finding container b63d3f445c17994573af3ec3b2c856d310183c0b717beb740a95493295435877: Status 404 returned error can't find the container with id b63d3f445c17994573af3ec3b2c856d310183c0b717beb740a95493295435877
Oct 31 03:11:31.646087 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:31.646055    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmdgv" event={"ID":"6e19afe0-f05b-4294-8d01-8d04cbb5b833","Type":"ContainerStarted","Data":"b63d3f445c17994573af3ec3b2c856d310183c0b717beb740a95493295435877"}
Oct 31 03:11:35.296987 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.296949    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-hlfv6"]
Oct 31 03:11:35.300961 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.300941    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hlfv6"
Oct 31 03:11:35.315507 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.315476    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4zlf\" (UniqueName: \"kubernetes.io/projected/89cdc3bb-4e7a-4fd8-9461-76cd0c789dec-kube-api-access-k4zlf\") pod \"cert-manager-597b95cb84-hlfv6\" (UID: \"89cdc3bb-4e7a-4fd8-9461-76cd0c789dec\") " pod="cert-manager/cert-manager-597b95cb84-hlfv6"
Oct 31 03:11:35.315658 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.315527    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/89cdc3bb-4e7a-4fd8-9461-76cd0c789dec-bound-sa-token\") pod \"cert-manager-597b95cb84-hlfv6\" (UID: \"89cdc3bb-4e7a-4fd8-9461-76cd0c789dec\") " pod="cert-manager/cert-manager-597b95cb84-hlfv6"
Oct 31 03:11:35.397351 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.397323    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-95952\""
Oct 31 03:11:35.398298 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.398276    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hlfv6"]
Oct 31 03:11:35.416408 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.416377    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4zlf\" (UniqueName: \"kubernetes.io/projected/89cdc3bb-4e7a-4fd8-9461-76cd0c789dec-kube-api-access-k4zlf\") pod \"cert-manager-597b95cb84-hlfv6\" (UID: \"89cdc3bb-4e7a-4fd8-9461-76cd0c789dec\") " pod="cert-manager/cert-manager-597b95cb84-hlfv6"
Oct 31 03:11:35.416566 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.416428    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/89cdc3bb-4e7a-4fd8-9461-76cd0c789dec-bound-sa-token\") pod \"cert-manager-597b95cb84-hlfv6\" (UID: \"89cdc3bb-4e7a-4fd8-9461-76cd0c789dec\") " pod="cert-manager/cert-manager-597b95cb84-hlfv6"
Oct 31 03:11:35.476747 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.476711    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/89cdc3bb-4e7a-4fd8-9461-76cd0c789dec-bound-sa-token\") pod \"cert-manager-597b95cb84-hlfv6\" (UID: \"89cdc3bb-4e7a-4fd8-9461-76cd0c789dec\") " pod="cert-manager/cert-manager-597b95cb84-hlfv6"
Oct 31 03:11:35.480098 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.480077    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4zlf\" (UniqueName: \"kubernetes.io/projected/89cdc3bb-4e7a-4fd8-9461-76cd0c789dec-kube-api-access-k4zlf\") pod \"cert-manager-597b95cb84-hlfv6\" (UID: \"89cdc3bb-4e7a-4fd8-9461-76cd0c789dec\") " pod="cert-manager/cert-manager-597b95cb84-hlfv6"
Oct 31 03:11:35.613048 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.613010    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hlfv6"
Oct 31 03:11:35.668719 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.668678    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmdgv" event={"ID":"6e19afe0-f05b-4294-8d01-8d04cbb5b833","Type":"ContainerStarted","Data":"dcbbe7295fa7769665b85b2aa8aebdf3766bfd61c901acc1d5fdf8b081fd7e71"}
Oct 31 03:11:35.668940 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.668832    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmdgv"
Oct 31 03:11:35.778793 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.778736    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmdgv" podStartSLOduration=1.743400253 podStartE2EDuration="5.778719788s" totalImagesPullingTime="4.035319535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:35.777921241 +0000 UTC m=+237.338400971" watchObservedRunningTime="2026-10-31 03:11:35.778719788 +0000 UTC m=+237.339199513"
Oct 31 03:11:35.996143 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:35.996038    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hlfv6"]
Oct 31 03:11:36.005124 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:11:36.005091    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89cdc3bb_4e7a_4fd8_9461_76cd0c789dec.slice/crio-1286bf327963ded58269bc64f880ddb1adfda7b4def6be9cc7cd853cee28ff02 WatchSource:0}: Error finding container 1286bf327963ded58269bc64f880ddb1adfda7b4def6be9cc7cd853cee28ff02: Status 404 returned error can't find the container with id 1286bf327963ded58269bc64f880ddb1adfda7b4def6be9cc7cd853cee28ff02
Oct 31 03:11:36.582897 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.582856    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"]
Oct 31 03:11:36.586337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.586315    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"
Oct 31 03:11:36.626673 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.626632    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4e3fc57-e3fd-4a86-954c-70cf7062810f-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-f4q4d\" (UID: \"b4e3fc57-e3fd-4a86-954c-70cf7062810f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"
Oct 31 03:11:36.627088 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.627053    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xrrc\" (UniqueName: \"kubernetes.io/projected/b4e3fc57-e3fd-4a86-954c-70cf7062810f-kube-api-access-2xrrc\") pod \"kuadrant-console-plugin-66c4fb575d-f4q4d\" (UID: \"b4e3fc57-e3fd-4a86-954c-70cf7062810f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"
Oct 31 03:11:36.627215 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.627167    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e3fc57-e3fd-4a86-954c-70cf7062810f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-f4q4d\" (UID: \"b4e3fc57-e3fd-4a86-954c-70cf7062810f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"
Oct 31 03:11:36.638108 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.638079    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 03:11:36.638108 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.638080    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 03:11:36.638279 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.638193    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4298d\""
Oct 31 03:11:36.646504 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.646475    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"]
Oct 31 03:11:36.674943 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.674905    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hlfv6" event={"ID":"89cdc3bb-4e7a-4fd8-9461-76cd0c789dec","Type":"ContainerStarted","Data":"ef9b67f1079348c351086ef4a5badd0b34f6773a6016e908f9d05a206cc8835b"}
Oct 31 03:11:36.675136 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.674955    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hlfv6" event={"ID":"89cdc3bb-4e7a-4fd8-9461-76cd0c789dec","Type":"ContainerStarted","Data":"1286bf327963ded58269bc64f880ddb1adfda7b4def6be9cc7cd853cee28ff02"}
Oct 31 03:11:36.728617 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.728573    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xrrc\" (UniqueName: \"kubernetes.io/projected/b4e3fc57-e3fd-4a86-954c-70cf7062810f-kube-api-access-2xrrc\") pod \"kuadrant-console-plugin-66c4fb575d-f4q4d\" (UID: \"b4e3fc57-e3fd-4a86-954c-70cf7062810f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"
Oct 31 03:11:36.728838 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.728657    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e3fc57-e3fd-4a86-954c-70cf7062810f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-f4q4d\" (UID: \"b4e3fc57-e3fd-4a86-954c-70cf7062810f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"
Oct 31 03:11:36.728838 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.728688    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4e3fc57-e3fd-4a86-954c-70cf7062810f-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-f4q4d\" (UID: \"b4e3fc57-e3fd-4a86-954c-70cf7062810f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"
Oct 31 03:11:36.729475 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.729452    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4e3fc57-e3fd-4a86-954c-70cf7062810f-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-f4q4d\" (UID: \"b4e3fc57-e3fd-4a86-954c-70cf7062810f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"
Oct 31 03:11:36.731200 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.731180    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e3fc57-e3fd-4a86-954c-70cf7062810f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-f4q4d\" (UID: \"b4e3fc57-e3fd-4a86-954c-70cf7062810f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"
Oct 31 03:11:36.782945 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.782907    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xrrc\" (UniqueName: \"kubernetes.io/projected/b4e3fc57-e3fd-4a86-954c-70cf7062810f-kube-api-access-2xrrc\") pod \"kuadrant-console-plugin-66c4fb575d-f4q4d\" (UID: \"b4e3fc57-e3fd-4a86-954c-70cf7062810f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"
Oct 31 03:11:36.801757 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.801706    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-hlfv6" podStartSLOduration=1.8016905539999999 podStartE2EDuration="1.801690554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:36.801440091 +0000 UTC m=+238.361919822" watchObservedRunningTime="2026-10-31 03:11:36.801690554 +0000 UTC m=+238.362170282"
Oct 31 03:11:36.898094 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:36.898004    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"
Oct 31 03:11:37.261460 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:37.261381    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d"]
Oct 31 03:11:37.264661 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:11:37.264632    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4e3fc57_e3fd_4a86_954c_70cf7062810f.slice/crio-59a1b5fd9ddf040c05f29088708fdb9440f1408e461702597e4f0d081e0b8a47 WatchSource:0}: Error finding container 59a1b5fd9ddf040c05f29088708fdb9440f1408e461702597e4f0d081e0b8a47: Status 404 returned error can't find the container with id 59a1b5fd9ddf040c05f29088708fdb9440f1408e461702597e4f0d081e0b8a47
Oct 31 03:11:37.683777 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:37.683739    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d" event={"ID":"b4e3fc57-e3fd-4a86-954c-70cf7062810f","Type":"ContainerStarted","Data":"59a1b5fd9ddf040c05f29088708fdb9440f1408e461702597e4f0d081e0b8a47"}
Oct 31 03:11:44.071963 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.071861    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"]
Oct 31 03:11:44.075415 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.075395    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"
Oct 31 03:11:44.130901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.130850    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 03:11:44.178153 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.178127    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 03:11:44.181598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.181577    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 03:11:44.181866 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.181851    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-csxjl\""
Oct 31 03:11:44.181957 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.181860    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 03:11:44.182651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.182627    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"]
Oct 31 03:11:44.200454 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.200382    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ec9b273-54b7-49b5-bd15-09d2230e7c3c-tmp\") pod \"kube-auth-proxy-cb6b5fd89-w7s8t\" (UID: \"5ec9b273-54b7-49b5-bd15-09d2230e7c3c\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"
Oct 31 03:11:44.200454 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.200442    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ec9b273-54b7-49b5-bd15-09d2230e7c3c-tls-certs\") pod \"kube-auth-proxy-cb6b5fd89-w7s8t\" (UID: \"5ec9b273-54b7-49b5-bd15-09d2230e7c3c\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"
Oct 31 03:11:44.200702 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.200475    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxzz\" (UniqueName: \"kubernetes.io/projected/5ec9b273-54b7-49b5-bd15-09d2230e7c3c-kube-api-access-hzxzz\") pod \"kube-auth-proxy-cb6b5fd89-w7s8t\" (UID: \"5ec9b273-54b7-49b5-bd15-09d2230e7c3c\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"
Oct 31 03:11:44.301517 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.301475    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ec9b273-54b7-49b5-bd15-09d2230e7c3c-tmp\") pod \"kube-auth-proxy-cb6b5fd89-w7s8t\" (UID: \"5ec9b273-54b7-49b5-bd15-09d2230e7c3c\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"
Oct 31 03:11:44.301682 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.301534    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ec9b273-54b7-49b5-bd15-09d2230e7c3c-tls-certs\") pod \"kube-auth-proxy-cb6b5fd89-w7s8t\" (UID: \"5ec9b273-54b7-49b5-bd15-09d2230e7c3c\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"
Oct 31 03:11:44.301682 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.301560    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxzz\" (UniqueName: \"kubernetes.io/projected/5ec9b273-54b7-49b5-bd15-09d2230e7c3c-kube-api-access-hzxzz\") pod \"kube-auth-proxy-cb6b5fd89-w7s8t\" (UID: \"5ec9b273-54b7-49b5-bd15-09d2230e7c3c\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"
Oct 31 03:11:44.303947 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.303917    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ec9b273-54b7-49b5-bd15-09d2230e7c3c-tmp\") pod \"kube-auth-proxy-cb6b5fd89-w7s8t\" (UID: \"5ec9b273-54b7-49b5-bd15-09d2230e7c3c\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"
Oct 31 03:11:44.304205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.304185    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ec9b273-54b7-49b5-bd15-09d2230e7c3c-tls-certs\") pod \"kube-auth-proxy-cb6b5fd89-w7s8t\" (UID: \"5ec9b273-54b7-49b5-bd15-09d2230e7c3c\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"
Oct 31 03:11:44.362938 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.362901    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxzz\" (UniqueName: \"kubernetes.io/projected/5ec9b273-54b7-49b5-bd15-09d2230e7c3c-kube-api-access-hzxzz\") pod \"kube-auth-proxy-cb6b5fd89-w7s8t\" (UID: \"5ec9b273-54b7-49b5-bd15-09d2230e7c3c\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"
Oct 31 03:11:44.387855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.387805    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"
Oct 31 03:11:44.721742 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.721656    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d" event={"ID":"b4e3fc57-e3fd-4a86-954c-70cf7062810f","Type":"ContainerStarted","Data":"00394425f0b9786fa5b5d15b7ae13f828d2f3ea0dec02625d1d2ffdad7e089d0"}
Oct 31 03:11:44.763903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.763841    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t"]
Oct 31 03:11:44.767373 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:11:44.767336    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec9b273_54b7_49b5_bd15_09d2230e7c3c.slice/crio-0a7c803b60a5888c877eed5c5ab78973c27e073fb162b0937f516823f50a074f WatchSource:0}: Error finding container 0a7c803b60a5888c877eed5c5ab78973c27e073fb162b0937f516823f50a074f: Status 404 returned error can't find the container with id 0a7c803b60a5888c877eed5c5ab78973c27e073fb162b0937f516823f50a074f
Oct 31 03:11:44.873490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:44.873389    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f4q4d" podStartSLOduration=2.428642822 podStartE2EDuration="8.873370681s" totalImagesPullingTime="6.444727859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:44.871099547 +0000 UTC m=+246.431579280" watchObservedRunningTime="2026-10-31 03:11:44.873370681 +0000 UTC m=+246.433850414"
Oct 31 03:11:45.727560 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:45.727514    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t" event={"ID":"5ec9b273-54b7-49b5-bd15-09d2230e7c3c","Type":"ContainerStarted","Data":"0a7c803b60a5888c877eed5c5ab78973c27e073fb162b0937f516823f50a074f"}
Oct 31 03:11:46.185447 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:11:46.185410    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:11:46.296337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:46.296296    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"]
Oct 31 03:11:46.300083 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:46.300051    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"
Oct 31 03:11:46.359705 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:46.359661    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 03:11:46.366177 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:46.366135    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"]
Oct 31 03:11:46.421095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:46.421057    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfbx\" (UniqueName: \"kubernetes.io/projected/e50a2998-9dba-440c-9be2-4eb990ff9aab-kube-api-access-kvfbx\") pod \"limitador-limitador-74dfcfb894-bvlnb\" (UID: \"e50a2998-9dba-440c-9be2-4eb990ff9aab\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"
Oct 31 03:11:46.421265 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:46.421116    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e50a2998-9dba-440c-9be2-4eb990ff9aab-config-file\") pod \"limitador-limitador-74dfcfb894-bvlnb\" (UID: \"e50a2998-9dba-440c-9be2-4eb990ff9aab\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"
Oct 31 03:11:46.522713 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:46.522623    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfbx\" (UniqueName: \"kubernetes.io/projected/e50a2998-9dba-440c-9be2-4eb990ff9aab-kube-api-access-kvfbx\") pod \"limitador-limitador-74dfcfb894-bvlnb\" (UID: \"e50a2998-9dba-440c-9be2-4eb990ff9aab\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"
Oct 31 03:11:46.522713 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:46.522671    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e50a2998-9dba-440c-9be2-4eb990ff9aab-config-file\") pod \"limitador-limitador-74dfcfb894-bvlnb\" (UID: \"e50a2998-9dba-440c-9be2-4eb990ff9aab\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"
Oct 31 03:11:46.523432 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:46.523401    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e50a2998-9dba-440c-9be2-4eb990ff9aab-config-file\") pod \"limitador-limitador-74dfcfb894-bvlnb\" (UID: \"e50a2998-9dba-440c-9be2-4eb990ff9aab\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"
Oct 31 03:11:46.541972 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:46.541825    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"]
Oct 31 03:11:46.579423 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:46.579395    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfbx\" (UniqueName: \"kubernetes.io/projected/e50a2998-9dba-440c-9be2-4eb990ff9aab-kube-api-access-kvfbx\") pod \"limitador-limitador-74dfcfb894-bvlnb\" (UID: \"e50a2998-9dba-440c-9be2-4eb990ff9aab\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"
Oct 31 03:11:46.618500 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:46.618468    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"
Oct 31 03:11:46.679263 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:46.678800    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmdgv"
Oct 31 03:11:47.000845 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:47.000666    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"]
Oct 31 03:11:47.741521 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:47.741471    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb" event={"ID":"e50a2998-9dba-440c-9be2-4eb990ff9aab","Type":"ContainerStarted","Data":"3158933cd0610b757d7e28072a35b93db7f46fb4195bf63bcfcf5e43362a4cfa"}
Oct 31 03:11:48.747123 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:48.747086    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t" event={"ID":"5ec9b273-54b7-49b5-bd15-09d2230e7c3c","Type":"ContainerStarted","Data":"5f5d734af4a880b85184fc280591bc14163ae8abec0c4a2e5f4abeef6b4e1d29"}
Oct 31 03:11:48.863208 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:48.863149    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-w7s8t" podStartSLOduration=1.618447761 podStartE2EDuration="4.863132903s" totalImagesPullingTime="3.244685142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:48.862740184 +0000 UTC m=+250.423219918" watchObservedRunningTime="2026-10-31 03:11:48.863132903 +0000 UTC m=+250.423612634"
Oct 31 03:11:49.753653 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:49.753608    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb" event={"ID":"e50a2998-9dba-440c-9be2-4eb990ff9aab","Type":"ContainerStarted","Data":"4a8895f80007a21bb1f6c16f8cc48b8dc12fbd172de501cf3e8fad10614b7faf"}
Oct 31 03:11:49.754089 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:49.753690    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"
Oct 31 03:11:49.873360 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:11:49.873303    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb" podStartSLOduration=1.6026240999999999 podStartE2EDuration="3.873278252s" totalImagesPullingTime="2.270654152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:49.872430015 +0000 UTC m=+251.432909766" watchObservedRunningTime="2026-10-31 03:11:49.873278252 +0000 UTC m=+251.433757982"
Oct 31 03:12:00.759055 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:00.759020    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"
Oct 31 03:12:16.178330 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:12:16.178244    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:12:21.923547 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:21.923507    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ft5cj"]
Oct 31 03:12:21.943470 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:21.943439    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ft5cj"
Oct 31 03:12:21.965699 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:21.965658    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg7sc\" (UniqueName: \"kubernetes.io/projected/be516f66-b120-4105-829f-900f21b4be9c-kube-api-access-gg7sc\") pod \"authorino-56c6c9ddd5-ft5cj\" (UID: \"be516f66-b120-4105-829f-900f21b4be9c\") " pod="kuadrant-system/authorino-56c6c9ddd5-ft5cj"
Oct 31 03:12:21.984360 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:21.984326    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ft5cj"]
Oct 31 03:12:22.030557 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:22.030525    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5bjvt\""
Oct 31 03:12:22.066687 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:22.066648    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg7sc\" (UniqueName: \"kubernetes.io/projected/be516f66-b120-4105-829f-900f21b4be9c-kube-api-access-gg7sc\") pod \"authorino-56c6c9ddd5-ft5cj\" (UID: \"be516f66-b120-4105-829f-900f21b4be9c\") " pod="kuadrant-system/authorino-56c6c9ddd5-ft5cj"
Oct 31 03:12:22.121425 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:22.121389    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg7sc\" (UniqueName: \"kubernetes.io/projected/be516f66-b120-4105-829f-900f21b4be9c-kube-api-access-gg7sc\") pod \"authorino-56c6c9ddd5-ft5cj\" (UID: \"be516f66-b120-4105-829f-900f21b4be9c\") " pod="kuadrant-system/authorino-56c6c9ddd5-ft5cj"
Oct 31 03:12:22.254385 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:22.254284    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ft5cj"
Oct 31 03:12:22.630735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:22.630695    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ft5cj"]
Oct 31 03:12:22.639063 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:12:22.639032    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe516f66_b120_4105_829f_900f21b4be9c.slice/crio-096755a1a329fee2c73752b46a366d00cda9defad2571f8701f7bb6125c1dab8 WatchSource:0}: Error finding container 096755a1a329fee2c73752b46a366d00cda9defad2571f8701f7bb6125c1dab8: Status 404 returned error can't find the container with id 096755a1a329fee2c73752b46a366d00cda9defad2571f8701f7bb6125c1dab8
Oct 31 03:12:22.900584 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:22.900488    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ft5cj" event={"ID":"be516f66-b120-4105-829f-900f21b4be9c","Type":"ContainerStarted","Data":"096755a1a329fee2c73752b46a366d00cda9defad2571f8701f7bb6125c1dab8"}
Oct 31 03:12:25.916156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:25.916113    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ft5cj" event={"ID":"be516f66-b120-4105-829f-900f21b4be9c","Type":"ContainerStarted","Data":"6b7270163c781a4f7da4db1aef7df28d8406755ca777d3b155e45a320f5e73c9"}
Oct 31 03:12:26.023547 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:26.023493    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-ft5cj" podStartSLOduration=2.091988681 podStartE2EDuration="5.023480132s" totalImagesPullingTime="2.931491451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:26.022607031 +0000 UTC m=+287.583086762" watchObservedRunningTime="2026-10-31 03:12:26.023480132 +0000 UTC m=+287.583959913"
Oct 31 03:12:33.072713 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.072673    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"]
Oct 31 03:12:33.073242 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.072969    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb" podUID="e50a2998-9dba-440c-9be2-4eb990ff9aab" containerName="limitador" containerID="cri-o://4a8895f80007a21bb1f6c16f8cc48b8dc12fbd172de501cf3e8fad10614b7faf" gracePeriod=30
Oct 31 03:12:33.622428 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.622400    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"
Oct 31 03:12:33.674284 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.674208    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e50a2998-9dba-440c-9be2-4eb990ff9aab-config-file\" (UniqueName: \"kubernetes.io/configmap/e50a2998-9dba-440c-9be2-4eb990ff9aab-config-file\") pod \"e50a2998-9dba-440c-9be2-4eb990ff9aab\" (UID: \"e50a2998-9dba-440c-9be2-4eb990ff9aab\") "
Oct 31 03:12:33.674414 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.674297    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e50a2998-9dba-440c-9be2-4eb990ff9aab-kube-api-access-kvfbx\" (UniqueName: \"kubernetes.io/projected/e50a2998-9dba-440c-9be2-4eb990ff9aab-kube-api-access-kvfbx\") pod \"e50a2998-9dba-440c-9be2-4eb990ff9aab\" (UID: \"e50a2998-9dba-440c-9be2-4eb990ff9aab\") "
Oct 31 03:12:33.674672 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.674645    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50a2998-9dba-440c-9be2-4eb990ff9aab-config-file" pod "e50a2998-9dba-440c-9be2-4eb990ff9aab" (UID: "e50a2998-9dba-440c-9be2-4eb990ff9aab"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:12:33.676546 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.676523    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50a2998-9dba-440c-9be2-4eb990ff9aab-kube-api-access-kvfbx" pod "e50a2998-9dba-440c-9be2-4eb990ff9aab" (UID: "e50a2998-9dba-440c-9be2-4eb990ff9aab"). InnerVolumeSpecName "kube-api-access-kvfbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:12:33.775692 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.775642    5415 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e50a2998-9dba-440c-9be2-4eb990ff9aab-config-file\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:12:33.775692 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.775686    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvfbx\" (UniqueName: \"kubernetes.io/projected/e50a2998-9dba-440c-9be2-4eb990ff9aab-kube-api-access-kvfbx\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:12:33.951034 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.950955    5415 generic.go:356] "Generic (PLEG): container finished" podID="e50a2998-9dba-440c-9be2-4eb990ff9aab" containerID="4a8895f80007a21bb1f6c16f8cc48b8dc12fbd172de501cf3e8fad10614b7faf" exitCode=0
Oct 31 03:12:33.951034 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.951003    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"
Oct 31 03:12:33.951034 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.951022    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb" event={"ID":"e50a2998-9dba-440c-9be2-4eb990ff9aab","Type":"ContainerDied","Data":"4a8895f80007a21bb1f6c16f8cc48b8dc12fbd172de501cf3e8fad10614b7faf"}
Oct 31 03:12:33.951262 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.951068    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-bvlnb" event={"ID":"e50a2998-9dba-440c-9be2-4eb990ff9aab","Type":"ContainerDied","Data":"3158933cd0610b757d7e28072a35b93db7f46fb4195bf63bcfcf5e43362a4cfa"}
Oct 31 03:12:33.951262 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.951089    5415 scope.go:122] "RemoveContainer" containerID="4a8895f80007a21bb1f6c16f8cc48b8dc12fbd172de501cf3e8fad10614b7faf"
Oct 31 03:12:33.961144 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.961121    5415 scope.go:122] "RemoveContainer" containerID="4a8895f80007a21bb1f6c16f8cc48b8dc12fbd172de501cf3e8fad10614b7faf"
Oct 31 03:12:33.961570 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:12:33.961505    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8895f80007a21bb1f6c16f8cc48b8dc12fbd172de501cf3e8fad10614b7faf\": container with ID starting with 4a8895f80007a21bb1f6c16f8cc48b8dc12fbd172de501cf3e8fad10614b7faf not found: ID does not exist" containerID="4a8895f80007a21bb1f6c16f8cc48b8dc12fbd172de501cf3e8fad10614b7faf"
Oct 31 03:12:33.961570 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:33.961545    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8895f80007a21bb1f6c16f8cc48b8dc12fbd172de501cf3e8fad10614b7faf"} err="failed to get container status \"4a8895f80007a21bb1f6c16f8cc48b8dc12fbd172de501cf3e8fad10614b7faf\": rpc error: code = NotFound desc = could not find container \"4a8895f80007a21bb1f6c16f8cc48b8dc12fbd172de501cf3e8fad10614b7faf\": container with ID starting with 4a8895f80007a21bb1f6c16f8cc48b8dc12fbd172de501cf3e8fad10614b7faf not found: ID does not exist"
Oct 31 03:12:34.116221 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:34.116186    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"]
Oct 31 03:12:34.163324 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:34.163291    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bvlnb"]
Oct 31 03:12:34.604582 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:34.604543    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e50a2998-9dba-440c-9be2-4eb990ff9aab" path="/var/lib/kubelet/pods/e50a2998-9dba-440c-9be2-4eb990ff9aab/volumes"
Oct 31 03:12:41.019833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.019797    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5"]
Oct 31 03:12:41.020237 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.020154    5415 cpu_manager.go:414] "containerMap: removing container" podUID="e50a2998-9dba-440c-9be2-4eb990ff9aab" containerName="limitador"
Oct 31 03:12:41.020237 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.020166    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e50a2998-9dba-440c-9be2-4eb990ff9aab" containerName="limitador"
Oct 31 03:12:41.020307 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.020239    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="e50a2998-9dba-440c-9be2-4eb990ff9aab" containerName="limitador"
Oct 31 03:12:41.024866 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.024845    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5"
Oct 31 03:12:41.074529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.074493    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:12:41.074529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.074505    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:12:41.080950 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.080915    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5"]
Oct 31 03:12:41.135013 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.134981    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 31 03:12:41.135013 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.135008    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-792hk\""
Oct 31 03:12:41.143408 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.143376    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr84w\" (UniqueName: \"kubernetes.io/projected/c934f25e-6d90-4da4-9594-009240e41070-kube-api-access-rr84w\") pod \"postgres-7cdbf6dc44-2hpz5\" (UID: \"c934f25e-6d90-4da4-9594-009240e41070\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5"
Oct 31 03:12:41.143569 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.143413    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c934f25e-6d90-4da4-9594-009240e41070-data\") pod \"postgres-7cdbf6dc44-2hpz5\" (UID: \"c934f25e-6d90-4da4-9594-009240e41070\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5"
Oct 31 03:12:41.244276 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.244235    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c934f25e-6d90-4da4-9594-009240e41070-data\") pod \"postgres-7cdbf6dc44-2hpz5\" (UID: \"c934f25e-6d90-4da4-9594-009240e41070\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5"
Oct 31 03:12:41.244457 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.244345    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr84w\" (UniqueName: \"kubernetes.io/projected/c934f25e-6d90-4da4-9594-009240e41070-kube-api-access-rr84w\") pod \"postgres-7cdbf6dc44-2hpz5\" (UID: \"c934f25e-6d90-4da4-9594-009240e41070\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5"
Oct 31 03:12:41.244653 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.244630    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c934f25e-6d90-4da4-9594-009240e41070-data\") pod \"postgres-7cdbf6dc44-2hpz5\" (UID: \"c934f25e-6d90-4da4-9594-009240e41070\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5"
Oct 31 03:12:41.300353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.300281    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr84w\" (UniqueName: \"kubernetes.io/projected/c934f25e-6d90-4da4-9594-009240e41070-kube-api-access-rr84w\") pod \"postgres-7cdbf6dc44-2hpz5\" (UID: \"c934f25e-6d90-4da4-9594-009240e41070\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5"
Oct 31 03:12:41.337179 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.337144    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5"
Oct 31 03:12:41.704800 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.704766    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5"]
Oct 31 03:12:41.708403 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:12:41.708368    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc934f25e_6d90_4da4_9594_009240e41070.slice/crio-2197dda412c637e3bf2407d01aa7f1e4d2f62ce071c7fc63f67dee4bf451e790 WatchSource:0}: Error finding container 2197dda412c637e3bf2407d01aa7f1e4d2f62ce071c7fc63f67dee4bf451e790: Status 404 returned error can't find the container with id 2197dda412c637e3bf2407d01aa7f1e4d2f62ce071c7fc63f67dee4bf451e790
Oct 31 03:12:41.709946 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.709929    5415 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:12:41.989744 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:41.989657    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5" event={"ID":"c934f25e-6d90-4da4-9594-009240e41070","Type":"ContainerStarted","Data":"2197dda412c637e3bf2407d01aa7f1e4d2f62ce071c7fc63f67dee4bf451e790"}
Oct 31 03:12:46.180435 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:12:46.180396    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:12:49.024614 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:49.024521    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5" event={"ID":"c934f25e-6d90-4da4-9594-009240e41070","Type":"ContainerStarted","Data":"4919b375f19668887d158312259f4c9a6ff10513ba7171e5a56b6ea06ccd83f6"}
Oct 31 03:12:49.024990 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:49.024713    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5"
Oct 31 03:12:49.134375 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:49.134317    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5" podStartSLOduration=2.116631256 podStartE2EDuration="9.134302307s" totalImagesPullingTime="7.017671051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:49.132645942 +0000 UTC m=+310.693125675" watchObservedRunningTime="2026-10-31 03:12:49.134302307 +0000 UTC m=+310.694782037"
Oct 31 03:12:55.058793 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:12:55.058762    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2hpz5"
Oct 31 03:13:16.008663 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.008629    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"]
Oct 31 03:13:16.012027 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.012005    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.025581 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.025554    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9dcba361-a1d8-4e62-843c-be420bd3fea8-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.025717 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.025611    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8q4q\" (UniqueName: \"kubernetes.io/projected/9dcba361-a1d8-4e62-843c-be420bd3fea8-kube-api-access-v8q4q\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.025717 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.025676    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9dcba361-a1d8-4e62-843c-be420bd3fea8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.025717 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.025713    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9dcba361-a1d8-4e62-843c-be420bd3fea8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.061747 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.061709    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 31 03:13:16.062016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.061996    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 31 03:13:16.108926 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.108865    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 31 03:13:16.109175 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.109160    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-rp7rr\""
Oct 31 03:13:16.109242 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.109202    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 03:13:16.109293 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.109262    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 03:13:16.110532 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.110506    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"]
Oct 31 03:13:16.126467 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.126434    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8q4q\" (UniqueName: \"kubernetes.io/projected/9dcba361-a1d8-4e62-843c-be420bd3fea8-kube-api-access-v8q4q\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.126655 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.126489    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9dcba361-a1d8-4e62-843c-be420bd3fea8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.126655 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.126534    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9dcba361-a1d8-4e62-843c-be420bd3fea8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.126655 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.126610    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9dcba361-a1d8-4e62-843c-be420bd3fea8-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.126810 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:13:16.126691    5415 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Oct 31 03:13:16.126810 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:13:16.126704    5415 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 31 03:13:16.126810 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:13:16.126784    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dcba361-a1d8-4e62-843c-be420bd3fea8-tls-cert podName:9dcba361-a1d8-4e62-843c-be420bd3fea8 nodeName:}" failed. No retries permitted until 2026-10-31 03:13:16.626767496 +0000 UTC m=+338.187247219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/9dcba361-a1d8-4e62-843c-be420bd3fea8-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-bk6x5" (UID: "9dcba361-a1d8-4e62-843c-be420bd3fea8") : secret "usage-logs-tenancy-proxy-tls" not found
Oct 31 03:13:16.126810 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:13:16.126801    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9dcba361-a1d8-4e62-843c-be420bd3fea8-proxy-source podName:9dcba361-a1d8-4e62-843c-be420bd3fea8 nodeName:}" failed. No retries permitted until 2026-10-31 03:13:16.626795401 +0000 UTC m=+338.187275109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/9dcba361-a1d8-4e62-843c-be420bd3fea8-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-bk6x5" (UID: "9dcba361-a1d8-4e62-843c-be420bd3fea8") : configmap "usage-tenancy-proxy-source" not found
Oct 31 03:13:16.127336 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.127313    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9dcba361-a1d8-4e62-843c-be420bd3fea8-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.174973 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:13:16.174934    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:13:16.182166 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.182138    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8q4q\" (UniqueName: \"kubernetes.io/projected/9dcba361-a1d8-4e62-843c-be420bd3fea8-kube-api-access-v8q4q\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.630249 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.630216    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9dcba361-a1d8-4e62-843c-be420bd3fea8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.630476 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.630266    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9dcba361-a1d8-4e62-843c-be420bd3fea8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.631190 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.631170    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9dcba361-a1d8-4e62-843c-be420bd3fea8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.632851 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.632828    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9dcba361-a1d8-4e62-843c-be420bd3fea8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bk6x5\" (UID: \"9dcba361-a1d8-4e62-843c-be420bd3fea8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.639916 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.639892    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qwnr5"]
Oct 31 03:13:16.643359 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.643340    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-qwnr5"
Oct 31 03:13:16.693714 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.693678    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2bzmz\""
Oct 31 03:13:16.693714 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.693705    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 31 03:13:16.702490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.702459    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qwnr5"]
Oct 31 03:13:16.730906 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.730849    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mllf2\" (UniqueName: \"kubernetes.io/projected/81594aa6-a125-4f16-9c62-5b3b6f24c6fb-kube-api-access-mllf2\") pod \"kserve-controller-manager-776cc66d47-qwnr5\" (UID: \"81594aa6-a125-4f16-9c62-5b3b6f24c6fb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qwnr5"
Oct 31 03:13:16.731122 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.730999    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81594aa6-a125-4f16-9c62-5b3b6f24c6fb-cert\") pod \"kserve-controller-manager-776cc66d47-qwnr5\" (UID: \"81594aa6-a125-4f16-9c62-5b3b6f24c6fb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qwnr5"
Oct 31 03:13:16.749505 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.749461    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"]
Oct 31 03:13:16.752824 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.752798    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"
Oct 31 03:13:16.803376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.803343    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gpq8j\""
Oct 31 03:13:16.803598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.803397    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 31 03:13:16.812737 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.812704    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"]
Oct 31 03:13:16.832102 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.832068    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sb49s\" (UID: \"180d2893-b9e6-4dc1-9bfd-00ca15cd286a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"
Oct 31 03:13:16.832289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.832153    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81594aa6-a125-4f16-9c62-5b3b6f24c6fb-cert\") pod \"kserve-controller-manager-776cc66d47-qwnr5\" (UID: \"81594aa6-a125-4f16-9c62-5b3b6f24c6fb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qwnr5"
Oct 31 03:13:16.832289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.832205    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6p4\" (UniqueName: \"kubernetes.io/projected/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-kube-api-access-cf6p4\") pod \"llmisvc-controller-manager-869d5dcc5d-sb49s\" (UID: \"180d2893-b9e6-4dc1-9bfd-00ca15cd286a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"
Oct 31 03:13:16.832289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.832235    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mllf2\" (UniqueName: \"kubernetes.io/projected/81594aa6-a125-4f16-9c62-5b3b6f24c6fb-kube-api-access-mllf2\") pod \"kserve-controller-manager-776cc66d47-qwnr5\" (UID: \"81594aa6-a125-4f16-9c62-5b3b6f24c6fb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qwnr5"
Oct 31 03:13:16.834633 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.834615    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81594aa6-a125-4f16-9c62-5b3b6f24c6fb-cert\") pod \"kserve-controller-manager-776cc66d47-qwnr5\" (UID: \"81594aa6-a125-4f16-9c62-5b3b6f24c6fb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qwnr5"
Oct 31 03:13:16.860390 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.860354    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4m5hw"]
Oct 31 03:13:16.864220 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.864194    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4m5hw"
Oct 31 03:13:16.887096 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.887026    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mllf2\" (UniqueName: \"kubernetes.io/projected/81594aa6-a125-4f16-9c62-5b3b6f24c6fb-kube-api-access-mllf2\") pod \"kserve-controller-manager-776cc66d47-qwnr5\" (UID: \"81594aa6-a125-4f16-9c62-5b3b6f24c6fb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qwnr5"
Oct 31 03:13:16.914437 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.914409    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 31 03:13:16.914619 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.914539    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-vr8d6\""
Oct 31 03:13:16.924069 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.924042    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:16.924190 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.924077    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4m5hw"]
Oct 31 03:13:16.933429 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.933403    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n4qk\" (UniqueName: \"kubernetes.io/projected/35e1e5d7-2240-4292-9f78-a9676f67e28a-kube-api-access-6n4qk\") pod \"model-serving-api-75bd89fdfc-4m5hw\" (UID: \"35e1e5d7-2240-4292-9f78-a9676f67e28a\") " pod="opendatahub/model-serving-api-75bd89fdfc-4m5hw"
Oct 31 03:13:16.933580 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.933467    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35e1e5d7-2240-4292-9f78-a9676f67e28a-tls-certs\") pod \"model-serving-api-75bd89fdfc-4m5hw\" (UID: \"35e1e5d7-2240-4292-9f78-a9676f67e28a\") " pod="opendatahub/model-serving-api-75bd89fdfc-4m5hw"
Oct 31 03:13:16.933580 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.933544    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6p4\" (UniqueName: \"kubernetes.io/projected/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-kube-api-access-cf6p4\") pod \"llmisvc-controller-manager-869d5dcc5d-sb49s\" (UID: \"180d2893-b9e6-4dc1-9bfd-00ca15cd286a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"
Oct 31 03:13:16.933907 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.933870    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sb49s\" (UID: \"180d2893-b9e6-4dc1-9bfd-00ca15cd286a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"
Oct 31 03:13:16.936726 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.936703    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sb49s\" (UID: \"180d2893-b9e6-4dc1-9bfd-00ca15cd286a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"
Oct 31 03:13:16.956755 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.956724    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-qwnr5"
Oct 31 03:13:16.967604 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.967572    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-hpjht"]
Oct 31 03:13:16.971234 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.971211    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-hpjht"
Oct 31 03:13:16.990517 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:16.990457    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6p4\" (UniqueName: \"kubernetes.io/projected/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-kube-api-access-cf6p4\") pod \"llmisvc-controller-manager-869d5dcc5d-sb49s\" (UID: \"180d2893-b9e6-4dc1-9bfd-00ca15cd286a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"
Oct 31 03:13:17.021197 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.021176    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 31 03:13:17.021593 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.021250    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bmb8h\""
Oct 31 03:13:17.021839 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.021822    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 31 03:13:17.031977 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.031921    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-hpjht"]
Oct 31 03:13:17.035169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.034641    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35e1e5d7-2240-4292-9f78-a9676f67e28a-tls-certs\") pod \"model-serving-api-75bd89fdfc-4m5hw\" (UID: \"35e1e5d7-2240-4292-9f78-a9676f67e28a\") " pod="opendatahub/model-serving-api-75bd89fdfc-4m5hw"
Oct 31 03:13:17.035169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.034684    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cb52f07-5218-4b28-896c-9b97e062ef44-cert\") pod \"odh-model-controller-769875db56-hpjht\" (UID: \"6cb52f07-5218-4b28-896c-9b97e062ef44\") " pod="opendatahub/odh-model-controller-769875db56-hpjht"
Oct 31 03:13:17.035169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.034852    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb52f07-5218-4b28-896c-9b97e062ef44-metrics-certs\") pod \"odh-model-controller-769875db56-hpjht\" (UID: \"6cb52f07-5218-4b28-896c-9b97e062ef44\") " pod="opendatahub/odh-model-controller-769875db56-hpjht"
Oct 31 03:13:17.035169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.035001    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n4qk\" (UniqueName: \"kubernetes.io/projected/35e1e5d7-2240-4292-9f78-a9676f67e28a-kube-api-access-6n4qk\") pod \"model-serving-api-75bd89fdfc-4m5hw\" (UID: \"35e1e5d7-2240-4292-9f78-a9676f67e28a\") " pod="opendatahub/model-serving-api-75bd89fdfc-4m5hw"
Oct 31 03:13:17.035169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.035040    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvmk\" (UniqueName: \"kubernetes.io/projected/6cb52f07-5218-4b28-896c-9b97e062ef44-kube-api-access-qrvmk\") pod \"odh-model-controller-769875db56-hpjht\" (UID: \"6cb52f07-5218-4b28-896c-9b97e062ef44\") " pod="opendatahub/odh-model-controller-769875db56-hpjht"
Oct 31 03:13:17.037731 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.037709    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35e1e5d7-2240-4292-9f78-a9676f67e28a-tls-certs\") pod \"model-serving-api-75bd89fdfc-4m5hw\" (UID: \"35e1e5d7-2240-4292-9f78-a9676f67e28a\") " pod="opendatahub/model-serving-api-75bd89fdfc-4m5hw"
Oct 31 03:13:17.065633 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.065595    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"
Oct 31 03:13:17.090398 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.090373    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n4qk\" (UniqueName: \"kubernetes.io/projected/35e1e5d7-2240-4292-9f78-a9676f67e28a-kube-api-access-6n4qk\") pod \"model-serving-api-75bd89fdfc-4m5hw\" (UID: \"35e1e5d7-2240-4292-9f78-a9676f67e28a\") " pod="opendatahub/model-serving-api-75bd89fdfc-4m5hw"
Oct 31 03:13:17.135558 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.135526    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cb52f07-5218-4b28-896c-9b97e062ef44-cert\") pod \"odh-model-controller-769875db56-hpjht\" (UID: \"6cb52f07-5218-4b28-896c-9b97e062ef44\") " pod="opendatahub/odh-model-controller-769875db56-hpjht"
Oct 31 03:13:17.135747 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.135570    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb52f07-5218-4b28-896c-9b97e062ef44-metrics-certs\") pod \"odh-model-controller-769875db56-hpjht\" (UID: \"6cb52f07-5218-4b28-896c-9b97e062ef44\") " pod="opendatahub/odh-model-controller-769875db56-hpjht"
Oct 31 03:13:17.135747 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.135636    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvmk\" (UniqueName: \"kubernetes.io/projected/6cb52f07-5218-4b28-896c-9b97e062ef44-kube-api-access-qrvmk\") pod \"odh-model-controller-769875db56-hpjht\" (UID: \"6cb52f07-5218-4b28-896c-9b97e062ef44\") " pod="opendatahub/odh-model-controller-769875db56-hpjht"
Oct 31 03:13:17.138445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.138353    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb52f07-5218-4b28-896c-9b97e062ef44-metrics-certs\") pod \"odh-model-controller-769875db56-hpjht\" (UID: \"6cb52f07-5218-4b28-896c-9b97e062ef44\") " pod="opendatahub/odh-model-controller-769875db56-hpjht"
Oct 31 03:13:17.138445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.138382    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cb52f07-5218-4b28-896c-9b97e062ef44-cert\") pod \"odh-model-controller-769875db56-hpjht\" (UID: \"6cb52f07-5218-4b28-896c-9b97e062ef44\") " pod="opendatahub/odh-model-controller-769875db56-hpjht"
Oct 31 03:13:17.178144 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.178102    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4m5hw"
Oct 31 03:13:17.190673 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.190647    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvmk\" (UniqueName: \"kubernetes.io/projected/6cb52f07-5218-4b28-896c-9b97e062ef44-kube-api-access-qrvmk\") pod \"odh-model-controller-769875db56-hpjht\" (UID: \"6cb52f07-5218-4b28-896c-9b97e062ef44\") " pod="opendatahub/odh-model-controller-769875db56-hpjht"
Oct 31 03:13:17.290321 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.290281    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-hpjht"
Oct 31 03:13:17.295695 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.295670    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"]
Oct 31 03:13:17.305338 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:13:17.303228    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcba361_a1d8_4e62_843c_be420bd3fea8.slice/crio-f07db9043af9cb078560a1fd401d8fb1ce7539ca322e12ed10888e5e4a7b3b8b WatchSource:0}: Error finding container f07db9043af9cb078560a1fd401d8fb1ce7539ca322e12ed10888e5e4a7b3b8b: Status 404 returned error can't find the container with id f07db9043af9cb078560a1fd401d8fb1ce7539ca322e12ed10888e5e4a7b3b8b
Oct 31 03:13:17.342158 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.342119    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qwnr5"]
Oct 31 03:13:17.350261 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:13:17.347434    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81594aa6_a125_4f16_9c62_5b3b6f24c6fb.slice/crio-b3d6d0fb2d8ca68c84131cce4e644ca62bd8ee51c37ba97225b960c847784c8f WatchSource:0}: Error finding container b3d6d0fb2d8ca68c84131cce4e644ca62bd8ee51c37ba97225b960c847784c8f: Status 404 returned error can't find the container with id b3d6d0fb2d8ca68c84131cce4e644ca62bd8ee51c37ba97225b960c847784c8f
Oct 31 03:13:17.430159 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.429920    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"]
Oct 31 03:13:17.432928 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:13:17.432901    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod180d2893_b9e6_4dc1_9bfd_00ca15cd286a.slice/crio-995eb9d37b394fe3f62a3774226dca2df57a2e2f9a26e10a13db161911f44826 WatchSource:0}: Error finding container 995eb9d37b394fe3f62a3774226dca2df57a2e2f9a26e10a13db161911f44826: Status 404 returned error can't find the container with id 995eb9d37b394fe3f62a3774226dca2df57a2e2f9a26e10a13db161911f44826
Oct 31 03:13:17.547334 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.547299    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4m5hw"]
Oct 31 03:13:17.550508 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:13:17.550476    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e1e5d7_2240_4292_9f78_a9676f67e28a.slice/crio-3e3d0d3b552cc5b407c9e4faab1721ac3b23a84dc70443cd76513191ce5089de WatchSource:0}: Error finding container 3e3d0d3b552cc5b407c9e4faab1721ac3b23a84dc70443cd76513191ce5089de: Status 404 returned error can't find the container with id 3e3d0d3b552cc5b407c9e4faab1721ac3b23a84dc70443cd76513191ce5089de
Oct 31 03:13:17.665668 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:17.665634    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-hpjht"]
Oct 31 03:13:17.669297 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:13:17.669268    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb52f07_5218_4b28_896c_9b97e062ef44.slice/crio-1b702dc0508cec3fc35aed537ee5dedf76c4823d9398183216c8da6a80612b8f WatchSource:0}: Error finding container 1b702dc0508cec3fc35aed537ee5dedf76c4823d9398183216c8da6a80612b8f: Status 404 returned error can't find the container with id 1b702dc0508cec3fc35aed537ee5dedf76c4823d9398183216c8da6a80612b8f
Oct 31 03:13:18.150856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:18.150820    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-qwnr5" event={"ID":"81594aa6-a125-4f16-9c62-5b3b6f24c6fb","Type":"ContainerStarted","Data":"b3d6d0fb2d8ca68c84131cce4e644ca62bd8ee51c37ba97225b960c847784c8f"}
Oct 31 03:13:18.151925 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:18.151902    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-hpjht" event={"ID":"6cb52f07-5218-4b28-896c-9b97e062ef44","Type":"ContainerStarted","Data":"1b702dc0508cec3fc35aed537ee5dedf76c4823d9398183216c8da6a80612b8f"}
Oct 31 03:13:18.152930 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:18.152909    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5" event={"ID":"9dcba361-a1d8-4e62-843c-be420bd3fea8","Type":"ContainerStarted","Data":"f07db9043af9cb078560a1fd401d8fb1ce7539ca322e12ed10888e5e4a7b3b8b"}
Oct 31 03:13:18.153921 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:18.153900    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s" event={"ID":"180d2893-b9e6-4dc1-9bfd-00ca15cd286a","Type":"ContainerStarted","Data":"995eb9d37b394fe3f62a3774226dca2df57a2e2f9a26e10a13db161911f44826"}
Oct 31 03:13:18.154966 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:18.154946    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4m5hw" event={"ID":"35e1e5d7-2240-4292-9f78-a9676f67e28a","Type":"ContainerStarted","Data":"3e3d0d3b552cc5b407c9e4faab1721ac3b23a84dc70443cd76513191ce5089de"}
Oct 31 03:13:21.186415 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:21.186366    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4m5hw" event={"ID":"35e1e5d7-2240-4292-9f78-a9676f67e28a","Type":"ContainerStarted","Data":"6e6cf4d782f7cc45a14148e34a4f8db4119b71365bfff058d14678496e5ee23d"}
Oct 31 03:13:21.187245 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:21.187224    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-4m5hw"
Oct 31 03:13:21.300317 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:21.300255    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-4m5hw" podStartSLOduration=2.153233995 podStartE2EDuration="5.300234464s" totalImagesPullingTime="3.147000469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:21.299028143 +0000 UTC m=+342.859507874" watchObservedRunningTime="2026-10-31 03:13:21.300234464 +0000 UTC m=+342.860714195"
Oct 31 03:13:22.198807 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:22.198745    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s" event={"ID":"180d2893-b9e6-4dc1-9bfd-00ca15cd286a","Type":"ContainerStarted","Data":"b2d6fc422fb685f06378aba7104a2ce7a7491af40944418f96d880b57c4e4282"}
Oct 31 03:13:22.199325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:22.198818    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"
Oct 31 03:13:23.204798 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:23.204763    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-qwnr5" event={"ID":"81594aa6-a125-4f16-9c62-5b3b6f24c6fb","Type":"ContainerStarted","Data":"4f950787e1b643a10bc12f0fcd16f4e1da1fe942bf31cd1ec79177f9a362132f"}
Oct 31 03:13:23.205213 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:23.205021    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-qwnr5"
Oct 31 03:13:23.206728 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:23.206686    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-hpjht" event={"ID":"6cb52f07-5218-4b28-896c-9b97e062ef44","Type":"ContainerStarted","Data":"5a23c01a450a832743f81a26ae5c3e47dde504ae935213931068229a554ec157"}
Oct 31 03:13:23.206869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:23.206741    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-hpjht"
Oct 31 03:13:23.325652 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:23.325576    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-qwnr5" podStartSLOduration=2.5887757000000002 podStartE2EDuration="7.325558034s" totalImagesPullingTime="4.736782334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:23.323180373 +0000 UTC m=+344.883660102" watchObservedRunningTime="2026-10-31 03:13:23.325558034 +0000 UTC m=+344.886037766"
Oct 31 03:13:23.327022 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:23.326970    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s" podStartSLOduration=2.935661882 podStartE2EDuration="7.326952991s" totalImagesPullingTime="4.391291109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:22.310530833 +0000 UTC m=+343.871010563" watchObservedRunningTime="2026-10-31 03:13:23.326952991 +0000 UTC m=+344.887432722"
Oct 31 03:13:23.432799 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:23.432715    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-hpjht" podStartSLOduration=2.596586975 podStartE2EDuration="7.432695474s" totalImagesPullingTime="4.836108499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:23.431699154 +0000 UTC m=+344.992178883" watchObservedRunningTime="2026-10-31 03:13:23.432695474 +0000 UTC m=+344.993175205"
Oct 31 03:13:33.211060 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:33.210984    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-4m5hw"
Oct 31 03:13:33.259717 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:33.259679    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5" event={"ID":"9dcba361-a1d8-4e62-843c-be420bd3fea8","Type":"ContainerStarted","Data":"38209c6114ed7ad4d817c09378a1c49a2372734f21e756bb7acdeb81f7b2f76f"}
Oct 31 03:13:33.259911 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:33.259753    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:33.430520 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:33.430464    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5" podStartSLOduration=2.882807262 podStartE2EDuration="18.430449365s" totalImagesPullingTime="15.547642103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:33.429866602 +0000 UTC m=+354.990346331" watchObservedRunningTime="2026-10-31 03:13:33.430449365 +0000 UTC m=+354.990929091"
Oct 31 03:13:34.214090 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:34.214057    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-hpjht"
Oct 31 03:13:44.271909 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:44.271856    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bk6x5"
Oct 31 03:13:46.175644 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:13:46.175560    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:13:54.213650 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:54.213618    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"
Oct 31 03:13:54.216420 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:54.216396    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-qwnr5"
Oct 31 03:13:59.403987 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:59.403952    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w9xp2"]
Oct 31 03:13:59.519960 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:59.519806    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w9xp2"]
Oct 31 03:13:59.519960 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:59.519934    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w9xp2"
Oct 31 03:13:59.547783 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:59.547745    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtvzk\" (UniqueName: \"kubernetes.io/projected/c7f368ab-e7eb-4880-92d6-b64efd381d1c-kube-api-access-wtvzk\") pod \"authorino-6f6944bcdd-w9xp2\" (UID: \"c7f368ab-e7eb-4880-92d6-b64efd381d1c\") " pod="kuadrant-system/authorino-6f6944bcdd-w9xp2"
Oct 31 03:13:59.649381 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:59.649320    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtvzk\" (UniqueName: \"kubernetes.io/projected/c7f368ab-e7eb-4880-92d6-b64efd381d1c-kube-api-access-wtvzk\") pod \"authorino-6f6944bcdd-w9xp2\" (UID: \"c7f368ab-e7eb-4880-92d6-b64efd381d1c\") " pod="kuadrant-system/authorino-6f6944bcdd-w9xp2"
Oct 31 03:13:59.707211 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:59.707122    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtvzk\" (UniqueName: \"kubernetes.io/projected/c7f368ab-e7eb-4880-92d6-b64efd381d1c-kube-api-access-wtvzk\") pod \"authorino-6f6944bcdd-w9xp2\" (UID: \"c7f368ab-e7eb-4880-92d6-b64efd381d1c\") " pod="kuadrant-system/authorino-6f6944bcdd-w9xp2"
Oct 31 03:13:59.762138 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:59.762102    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w9xp2"]
Oct 31 03:13:59.762328 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:59.762317    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w9xp2"
Oct 31 03:13:59.789278 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:59.789243    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74ff8459f7-x7rcs"]
Oct 31 03:13:59.792530 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:59.792504    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ff8459f7-x7rcs"
Oct 31 03:13:59.850308 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:59.850269    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl7gw\" (UniqueName: \"kubernetes.io/projected/180c394f-c5b1-47ce-b77e-e1d42a3bfd82-kube-api-access-wl7gw\") pod \"authorino-74ff8459f7-x7rcs\" (UID: \"180c394f-c5b1-47ce-b77e-e1d42a3bfd82\") " pod="kuadrant-system/authorino-74ff8459f7-x7rcs"
Oct 31 03:13:59.851049 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:59.851028    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74ff8459f7-x7rcs"]
Oct 31 03:13:59.950913 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:13:59.950843    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl7gw\" (UniqueName: \"kubernetes.io/projected/180c394f-c5b1-47ce-b77e-e1d42a3bfd82-kube-api-access-wl7gw\") pod \"authorino-74ff8459f7-x7rcs\" (UID: \"180c394f-c5b1-47ce-b77e-e1d42a3bfd82\") " pod="kuadrant-system/authorino-74ff8459f7-x7rcs"
Oct 31 03:14:00.004731 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.004645    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl7gw\" (UniqueName: \"kubernetes.io/projected/180c394f-c5b1-47ce-b77e-e1d42a3bfd82-kube-api-access-wl7gw\") pod \"authorino-74ff8459f7-x7rcs\" (UID: \"180c394f-c5b1-47ce-b77e-e1d42a3bfd82\") " pod="kuadrant-system/authorino-74ff8459f7-x7rcs"
Oct 31 03:14:00.105728 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.105694    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ff8459f7-x7rcs"
Oct 31 03:14:00.125778 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.125749    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w9xp2"]
Oct 31 03:14:00.129059 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:14:00.129028    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7f368ab_e7eb_4880_92d6_b64efd381d1c.slice/crio-77472c5c84083c83b311995972b241c85ecdc47c120251f5e40fc8798381a7bf WatchSource:0}: Error finding container 77472c5c84083c83b311995972b241c85ecdc47c120251f5e40fc8798381a7bf: Status 404 returned error can't find the container with id 77472c5c84083c83b311995972b241c85ecdc47c120251f5e40fc8798381a7bf
Oct 31 03:14:00.279588 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.279502    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74ff8459f7-x7rcs"]
Oct 31 03:14:00.347530 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.347494    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d44779cff-9cj59"]
Oct 31 03:14:00.351146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.351127    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d44779cff-9cj59"
Oct 31 03:14:00.353850 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.353825    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/037e56de-57bc-440c-a7d2-bb1f80aa0264-tls-cert\") pod \"authorino-d44779cff-9cj59\" (UID: \"037e56de-57bc-440c-a7d2-bb1f80aa0264\") " pod="kuadrant-system/authorino-d44779cff-9cj59"
Oct 31 03:14:00.353984 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.353859    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn4gr\" (UniqueName: \"kubernetes.io/projected/037e56de-57bc-440c-a7d2-bb1f80aa0264-kube-api-access-zn4gr\") pod \"authorino-d44779cff-9cj59\" (UID: \"037e56de-57bc-440c-a7d2-bb1f80aa0264\") " pod="kuadrant-system/authorino-d44779cff-9cj59"
Oct 31 03:14:00.381922 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.381872    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-w9xp2" event={"ID":"c7f368ab-e7eb-4880-92d6-b64efd381d1c","Type":"ContainerStarted","Data":"77472c5c84083c83b311995972b241c85ecdc47c120251f5e40fc8798381a7bf"}
Oct 31 03:14:00.400983 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.400938    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 03:14:00.409114 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.409085    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d44779cff-9cj59"]
Oct 31 03:14:00.454679 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.454641    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/037e56de-57bc-440c-a7d2-bb1f80aa0264-tls-cert\") pod \"authorino-d44779cff-9cj59\" (UID: \"037e56de-57bc-440c-a7d2-bb1f80aa0264\") " pod="kuadrant-system/authorino-d44779cff-9cj59"
Oct 31 03:14:00.454679 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.454683    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn4gr\" (UniqueName: \"kubernetes.io/projected/037e56de-57bc-440c-a7d2-bb1f80aa0264-kube-api-access-zn4gr\") pod \"authorino-d44779cff-9cj59\" (UID: \"037e56de-57bc-440c-a7d2-bb1f80aa0264\") " pod="kuadrant-system/authorino-d44779cff-9cj59"
Oct 31 03:14:00.457095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.457067    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/037e56de-57bc-440c-a7d2-bb1f80aa0264-tls-cert\") pod \"authorino-d44779cff-9cj59\" (UID: \"037e56de-57bc-440c-a7d2-bb1f80aa0264\") " pod="kuadrant-system/authorino-d44779cff-9cj59"
Oct 31 03:14:00.468183 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.468159    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74ff8459f7-x7rcs"]
Oct 31 03:14:00.470870 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:14:00.470838    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod180c394f_c5b1_47ce_b77e_e1d42a3bfd82.slice/crio-23681eae75abf19f0c6310413d908e51087a5ec2eb5fa7c35c7fe4b22716fbf1 WatchSource:0}: Error finding container 23681eae75abf19f0c6310413d908e51087a5ec2eb5fa7c35c7fe4b22716fbf1: Status 404 returned error can't find the container with id 23681eae75abf19f0c6310413d908e51087a5ec2eb5fa7c35c7fe4b22716fbf1
Oct 31 03:14:00.509619 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.509590    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn4gr\" (UniqueName: \"kubernetes.io/projected/037e56de-57bc-440c-a7d2-bb1f80aa0264-kube-api-access-zn4gr\") pod \"authorino-d44779cff-9cj59\" (UID: \"037e56de-57bc-440c-a7d2-bb1f80aa0264\") " pod="kuadrant-system/authorino-d44779cff-9cj59"
Oct 31 03:14:00.663492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:00.663455    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d44779cff-9cj59"
Oct 31 03:14:01.026618 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:01.026536    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d44779cff-9cj59"]
Oct 31 03:14:01.029503 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:14:01.029456    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod037e56de_57bc_440c_a7d2_bb1f80aa0264.slice/crio-82696d41069207dd3f3e32585cbc2b92791ff8cfbd7baa817162df7ab9982921 WatchSource:0}: Error finding container 82696d41069207dd3f3e32585cbc2b92791ff8cfbd7baa817162df7ab9982921: Status 404 returned error can't find the container with id 82696d41069207dd3f3e32585cbc2b92791ff8cfbd7baa817162df7ab9982921
Oct 31 03:14:01.387633 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:01.387593    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d44779cff-9cj59" event={"ID":"037e56de-57bc-440c-a7d2-bb1f80aa0264","Type":"ContainerStarted","Data":"82696d41069207dd3f3e32585cbc2b92791ff8cfbd7baa817162df7ab9982921"}
Oct 31 03:14:01.388752 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:01.388730    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ff8459f7-x7rcs" event={"ID":"180c394f-c5b1-47ce-b77e-e1d42a3bfd82","Type":"ContainerStarted","Data":"23681eae75abf19f0c6310413d908e51087a5ec2eb5fa7c35c7fe4b22716fbf1"}
Oct 31 03:14:02.394816 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.394775    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-w9xp2" event={"ID":"c7f368ab-e7eb-4880-92d6-b64efd381d1c","Type":"ContainerStarted","Data":"1812cf41decf821514017f61f86fa4b0607f39f9b76be5be4d5ab98791bec1e5"}
Oct 31 03:14:02.394816 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.394775    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-w9xp2" podUID="c7f368ab-e7eb-4880-92d6-b64efd381d1c" containerName="authorino" containerID="cri-o://1812cf41decf821514017f61f86fa4b0607f39f9b76be5be4d5ab98791bec1e5" gracePeriod=30
Oct 31 03:14:02.396515 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.396485    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ff8459f7-x7rcs" event={"ID":"180c394f-c5b1-47ce-b77e-e1d42a3bfd82","Type":"ContainerStarted","Data":"65506aa8db48216703594f9e67a683618758171de77d871e64fa6176ba021c54"}
Oct 31 03:14:02.396650 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.396580    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-74ff8459f7-x7rcs" podUID="180c394f-c5b1-47ce-b77e-e1d42a3bfd82" containerName="authorino" containerID="cri-o://65506aa8db48216703594f9e67a683618758171de77d871e64fa6176ba021c54" gracePeriod=30
Oct 31 03:14:02.504567 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.504518    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-w9xp2" podStartSLOduration=1.747305694 podStartE2EDuration="3.504502233s" totalImagesPullingTime="1.757196539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:02.504113597 +0000 UTC m=+384.064593328" watchObservedRunningTime="2026-10-31 03:14:02.504502233 +0000 UTC m=+384.064981963"
Oct 31 03:14:02.611425 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.611371    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-74ff8459f7-x7rcs" podStartSLOduration=2.063987124 podStartE2EDuration="3.611354876s" totalImagesPullingTime="1.547367752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:02.610041599 +0000 UTC m=+384.170521328" watchObservedRunningTime="2026-10-31 03:14:02.611354876 +0000 UTC m=+384.171834606"
Oct 31 03:14:02.661329 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.661300    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ff8459f7-x7rcs"
Oct 31 03:14:02.664670 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.664644    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w9xp2"
Oct 31 03:14:02.670580 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.670553    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f368ab-e7eb-4880-92d6-b64efd381d1c-kube-api-access-wtvzk\" (UniqueName: \"kubernetes.io/projected/c7f368ab-e7eb-4880-92d6-b64efd381d1c-kube-api-access-wtvzk\") pod \"c7f368ab-e7eb-4880-92d6-b64efd381d1c\" (UID: \"c7f368ab-e7eb-4880-92d6-b64efd381d1c\") "
Oct 31 03:14:02.670707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.670625    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/180c394f-c5b1-47ce-b77e-e1d42a3bfd82-kube-api-access-wl7gw\" (UniqueName: \"kubernetes.io/projected/180c394f-c5b1-47ce-b77e-e1d42a3bfd82-kube-api-access-wl7gw\") pod \"180c394f-c5b1-47ce-b77e-e1d42a3bfd82\" (UID: \"180c394f-c5b1-47ce-b77e-e1d42a3bfd82\") "
Oct 31 03:14:02.672749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.672720    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180c394f-c5b1-47ce-b77e-e1d42a3bfd82-kube-api-access-wl7gw" pod "180c394f-c5b1-47ce-b77e-e1d42a3bfd82" (UID: "180c394f-c5b1-47ce-b77e-e1d42a3bfd82"). InnerVolumeSpecName "kube-api-access-wl7gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:14:02.672859 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.672756    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f368ab-e7eb-4880-92d6-b64efd381d1c-kube-api-access-wtvzk" pod "c7f368ab-e7eb-4880-92d6-b64efd381d1c" (UID: "c7f368ab-e7eb-4880-92d6-b64efd381d1c"). InnerVolumeSpecName "kube-api-access-wtvzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:14:02.772296 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.772261    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl7gw\" (UniqueName: \"kubernetes.io/projected/180c394f-c5b1-47ce-b77e-e1d42a3bfd82-kube-api-access-wl7gw\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:14:02.772296 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:02.772292    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtvzk\" (UniqueName: \"kubernetes.io/projected/c7f368ab-e7eb-4880-92d6-b64efd381d1c-kube-api-access-wtvzk\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:14:03.407917 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.406678    5415 generic.go:356] "Generic (PLEG): container finished" podID="c7f368ab-e7eb-4880-92d6-b64efd381d1c" containerID="1812cf41decf821514017f61f86fa4b0607f39f9b76be5be4d5ab98791bec1e5" exitCode=0
Oct 31 03:14:03.407917 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.406838    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-w9xp2" event={"ID":"c7f368ab-e7eb-4880-92d6-b64efd381d1c","Type":"ContainerDied","Data":"1812cf41decf821514017f61f86fa4b0607f39f9b76be5be4d5ab98791bec1e5"}
Oct 31 03:14:03.407917 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.406894    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-w9xp2" event={"ID":"c7f368ab-e7eb-4880-92d6-b64efd381d1c","Type":"ContainerDied","Data":"77472c5c84083c83b311995972b241c85ecdc47c120251f5e40fc8798381a7bf"}
Oct 31 03:14:03.407917 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.406919    5415 scope.go:122] "RemoveContainer" containerID="1812cf41decf821514017f61f86fa4b0607f39f9b76be5be4d5ab98791bec1e5"
Oct 31 03:14:03.407917 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.407062    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w9xp2"
Oct 31 03:14:03.412021 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.411981    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d44779cff-9cj59" event={"ID":"037e56de-57bc-440c-a7d2-bb1f80aa0264","Type":"ContainerStarted","Data":"8cb23fe66066fee4191b6c6c20f7188d0898760d152eb8702a5c53d306339a54"}
Oct 31 03:14:03.417458 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.417410    5415 generic.go:356] "Generic (PLEG): container finished" podID="180c394f-c5b1-47ce-b77e-e1d42a3bfd82" containerID="65506aa8db48216703594f9e67a683618758171de77d871e64fa6176ba021c54" exitCode=0
Oct 31 03:14:03.417539 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.417493    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ff8459f7-x7rcs" event={"ID":"180c394f-c5b1-47ce-b77e-e1d42a3bfd82","Type":"ContainerDied","Data":"65506aa8db48216703594f9e67a683618758171de77d871e64fa6176ba021c54"}
Oct 31 03:14:03.417584 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.417537    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ff8459f7-x7rcs" event={"ID":"180c394f-c5b1-47ce-b77e-e1d42a3bfd82","Type":"ContainerDied","Data":"23681eae75abf19f0c6310413d908e51087a5ec2eb5fa7c35c7fe4b22716fbf1"}
Oct 31 03:14:03.417624 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.417599    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ff8459f7-x7rcs"
Oct 31 03:14:03.421462 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.421300    5415 scope.go:122] "RemoveContainer" containerID="1812cf41decf821514017f61f86fa4b0607f39f9b76be5be4d5ab98791bec1e5"
Oct 31 03:14:03.421704 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:14:03.421672    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1812cf41decf821514017f61f86fa4b0607f39f9b76be5be4d5ab98791bec1e5\": container with ID starting with 1812cf41decf821514017f61f86fa4b0607f39f9b76be5be4d5ab98791bec1e5 not found: ID does not exist" containerID="1812cf41decf821514017f61f86fa4b0607f39f9b76be5be4d5ab98791bec1e5"
Oct 31 03:14:03.421903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.421718    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1812cf41decf821514017f61f86fa4b0607f39f9b76be5be4d5ab98791bec1e5"} err="failed to get container status \"1812cf41decf821514017f61f86fa4b0607f39f9b76be5be4d5ab98791bec1e5\": rpc error: code = NotFound desc = could not find container \"1812cf41decf821514017f61f86fa4b0607f39f9b76be5be4d5ab98791bec1e5\": container with ID starting with 1812cf41decf821514017f61f86fa4b0607f39f9b76be5be4d5ab98791bec1e5 not found: ID does not exist"
Oct 31 03:14:03.421903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.421744    5415 scope.go:122] "RemoveContainer" containerID="65506aa8db48216703594f9e67a683618758171de77d871e64fa6176ba021c54"
Oct 31 03:14:03.433285 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.433251    5415 scope.go:122] "RemoveContainer" containerID="65506aa8db48216703594f9e67a683618758171de77d871e64fa6176ba021c54"
Oct 31 03:14:03.433663 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:14:03.433637    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65506aa8db48216703594f9e67a683618758171de77d871e64fa6176ba021c54\": container with ID starting with 65506aa8db48216703594f9e67a683618758171de77d871e64fa6176ba021c54 not found: ID does not exist" containerID="65506aa8db48216703594f9e67a683618758171de77d871e64fa6176ba021c54"
Oct 31 03:14:03.433761 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.433671    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65506aa8db48216703594f9e67a683618758171de77d871e64fa6176ba021c54"} err="failed to get container status \"65506aa8db48216703594f9e67a683618758171de77d871e64fa6176ba021c54\": rpc error: code = NotFound desc = could not find container \"65506aa8db48216703594f9e67a683618758171de77d871e64fa6176ba021c54\": container with ID starting with 65506aa8db48216703594f9e67a683618758171de77d871e64fa6176ba021c54 not found: ID does not exist"
Oct 31 03:14:03.569271 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.569227    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w9xp2"]
Oct 31 03:14:03.575362 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.575326    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w9xp2"]
Oct 31 03:14:03.679437 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.679357    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74ff8459f7-x7rcs"]
Oct 31 03:14:03.686117 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.686084    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74ff8459f7-x7rcs"]
Oct 31 03:14:03.798179 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.798110    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-d44779cff-9cj59" podStartSLOduration=2.144792614 podStartE2EDuration="3.798088466s" totalImagesPullingTime="1.653295852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:03.797350673 +0000 UTC m=+385.357830403" watchObservedRunningTime="2026-10-31 03:14:03.798088466 +0000 UTC m=+385.358568266"
Oct 31 03:14:03.823258 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.823218    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ft5cj"]
Oct 31 03:14:03.823792 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:03.823528    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-ft5cj" podUID="be516f66-b120-4105-829f-900f21b4be9c" containerName="authorino" containerID="cri-o://6b7270163c781a4f7da4db1aef7df28d8406755ca777d3b155e45a320f5e73c9" gracePeriod=30
Oct 31 03:14:04.071820 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.071793    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ft5cj"
Oct 31 03:14:04.084889 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.084853    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be516f66-b120-4105-829f-900f21b4be9c-kube-api-access-gg7sc\" (UniqueName: \"kubernetes.io/projected/be516f66-b120-4105-829f-900f21b4be9c-kube-api-access-gg7sc\") pod \"be516f66-b120-4105-829f-900f21b4be9c\" (UID: \"be516f66-b120-4105-829f-900f21b4be9c\") "
Oct 31 03:14:04.086854 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.086827    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be516f66-b120-4105-829f-900f21b4be9c-kube-api-access-gg7sc" pod "be516f66-b120-4105-829f-900f21b4be9c" (UID: "be516f66-b120-4105-829f-900f21b4be9c"). InnerVolumeSpecName "kube-api-access-gg7sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:14:04.186052 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.186018    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg7sc\" (UniqueName: \"kubernetes.io/projected/be516f66-b120-4105-829f-900f21b4be9c-kube-api-access-gg7sc\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:14:04.424219 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.424186    5415 generic.go:356] "Generic (PLEG): container finished" podID="be516f66-b120-4105-829f-900f21b4be9c" containerID="6b7270163c781a4f7da4db1aef7df28d8406755ca777d3b155e45a320f5e73c9" exitCode=0
Oct 31 03:14:04.424673 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.424234    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ft5cj"
Oct 31 03:14:04.424673 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.424273    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ft5cj" event={"ID":"be516f66-b120-4105-829f-900f21b4be9c","Type":"ContainerDied","Data":"6b7270163c781a4f7da4db1aef7df28d8406755ca777d3b155e45a320f5e73c9"}
Oct 31 03:14:04.424673 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.424314    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ft5cj" event={"ID":"be516f66-b120-4105-829f-900f21b4be9c","Type":"ContainerDied","Data":"096755a1a329fee2c73752b46a366d00cda9defad2571f8701f7bb6125c1dab8"}
Oct 31 03:14:04.424673 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.424332    5415 scope.go:122] "RemoveContainer" containerID="6b7270163c781a4f7da4db1aef7df28d8406755ca777d3b155e45a320f5e73c9"
Oct 31 03:14:04.433005 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.432987    5415 scope.go:122] "RemoveContainer" containerID="6b7270163c781a4f7da4db1aef7df28d8406755ca777d3b155e45a320f5e73c9"
Oct 31 03:14:04.433307 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:14:04.433291    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b7270163c781a4f7da4db1aef7df28d8406755ca777d3b155e45a320f5e73c9\": container with ID starting with 6b7270163c781a4f7da4db1aef7df28d8406755ca777d3b155e45a320f5e73c9 not found: ID does not exist" containerID="6b7270163c781a4f7da4db1aef7df28d8406755ca777d3b155e45a320f5e73c9"
Oct 31 03:14:04.433363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.433320    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b7270163c781a4f7da4db1aef7df28d8406755ca777d3b155e45a320f5e73c9"} err="failed to get container status \"6b7270163c781a4f7da4db1aef7df28d8406755ca777d3b155e45a320f5e73c9\": rpc error: code = NotFound desc = could not find container \"6b7270163c781a4f7da4db1aef7df28d8406755ca777d3b155e45a320f5e73c9\": container with ID starting with 6b7270163c781a4f7da4db1aef7df28d8406755ca777d3b155e45a320f5e73c9 not found: ID does not exist"
Oct 31 03:14:04.578315 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.578280    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ft5cj"]
Oct 31 03:14:04.581652 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.581624    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ft5cj"]
Oct 31 03:14:04.605071 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.605041    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="180c394f-c5b1-47ce-b77e-e1d42a3bfd82" path="/var/lib/kubelet/pods/180c394f-c5b1-47ce-b77e-e1d42a3bfd82/volumes"
Oct 31 03:14:04.605378 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.605366    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be516f66-b120-4105-829f-900f21b4be9c" path="/var/lib/kubelet/pods/be516f66-b120-4105-829f-900f21b4be9c/volumes"
Oct 31 03:14:04.605654 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:04.605642    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f368ab-e7eb-4880-92d6-b64efd381d1c" path="/var/lib/kubelet/pods/c7f368ab-e7eb-4880-92d6-b64efd381d1c/volumes"
Oct 31 03:14:16.175794 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:14:16.175756    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:14:24.909804 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.909765    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t"]
Oct 31 03:14:24.910211 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.910182    5415 cpu_manager.go:414] "containerMap: removing container" podUID="be516f66-b120-4105-829f-900f21b4be9c" containerName="authorino"
Oct 31 03:14:24.910211 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.910193    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be516f66-b120-4105-829f-900f21b4be9c" containerName="authorino"
Oct 31 03:14:24.910285 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.910213    5415 cpu_manager.go:414] "containerMap: removing container" podUID="c7f368ab-e7eb-4880-92d6-b64efd381d1c" containerName="authorino"
Oct 31 03:14:24.910285 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.910221    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f368ab-e7eb-4880-92d6-b64efd381d1c" containerName="authorino"
Oct 31 03:14:24.910285 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.910235    5415 cpu_manager.go:414] "containerMap: removing container" podUID="180c394f-c5b1-47ce-b77e-e1d42a3bfd82" containerName="authorino"
Oct 31 03:14:24.910285 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.910242    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180c394f-c5b1-47ce-b77e-e1d42a3bfd82" containerName="authorino"
Oct 31 03:14:24.910391 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.910313    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f368ab-e7eb-4880-92d6-b64efd381d1c" containerName="authorino"
Oct 31 03:14:24.910391 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.910322    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="180c394f-c5b1-47ce-b77e-e1d42a3bfd82" containerName="authorino"
Oct 31 03:14:24.910391 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.910331    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="be516f66-b120-4105-829f-900f21b4be9c" containerName="authorino"
Oct 31 03:14:24.912783 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.912764    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t"
Oct 31 03:14:24.972561 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.972527    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t"]
Oct 31 03:14:24.981834 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.981803    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7n6s\" (UniqueName: \"kubernetes.io/projected/e94e222c-4de2-4046-a659-0c92420cc03e-kube-api-access-j7n6s\") pod \"llmisvc-controller-manager-6466c554d8-cmm9t\" (UID: \"e94e222c-4de2-4046-a659-0c92420cc03e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t"
Oct 31 03:14:24.982009 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:24.981946    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e94e222c-4de2-4046-a659-0c92420cc03e-cert\") pod \"llmisvc-controller-manager-6466c554d8-cmm9t\" (UID: \"e94e222c-4de2-4046-a659-0c92420cc03e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t"
Oct 31 03:14:25.082587 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:25.082548    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7n6s\" (UniqueName: \"kubernetes.io/projected/e94e222c-4de2-4046-a659-0c92420cc03e-kube-api-access-j7n6s\") pod \"llmisvc-controller-manager-6466c554d8-cmm9t\" (UID: \"e94e222c-4de2-4046-a659-0c92420cc03e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t"
Oct 31 03:14:25.082773 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:25.082624    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e94e222c-4de2-4046-a659-0c92420cc03e-cert\") pod \"llmisvc-controller-manager-6466c554d8-cmm9t\" (UID: \"e94e222c-4de2-4046-a659-0c92420cc03e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t"
Oct 31 03:14:25.085019 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:25.084998    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e94e222c-4de2-4046-a659-0c92420cc03e-cert\") pod \"llmisvc-controller-manager-6466c554d8-cmm9t\" (UID: \"e94e222c-4de2-4046-a659-0c92420cc03e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t"
Oct 31 03:14:25.137930 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:25.137868    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7n6s\" (UniqueName: \"kubernetes.io/projected/e94e222c-4de2-4046-a659-0c92420cc03e-kube-api-access-j7n6s\") pod \"llmisvc-controller-manager-6466c554d8-cmm9t\" (UID: \"e94e222c-4de2-4046-a659-0c92420cc03e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t"
Oct 31 03:14:25.225027 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:25.224931    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t"
Oct 31 03:14:25.595812 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:25.595778    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t"]
Oct 31 03:14:25.598824 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:14:25.598783    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94e222c_4de2_4046_a659_0c92420cc03e.slice/crio-f60733e2b63440a02435ca09156dd54aec1c6a873ee6b7231c714a4f176a0bba WatchSource:0}: Error finding container f60733e2b63440a02435ca09156dd54aec1c6a873ee6b7231c714a4f176a0bba: Status 404 returned error can't find the container with id f60733e2b63440a02435ca09156dd54aec1c6a873ee6b7231c714a4f176a0bba
Oct 31 03:14:26.528324 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:26.528294    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t" event={"ID":"e94e222c-4de2-4046-a659-0c92420cc03e","Type":"ContainerStarted","Data":"9304df5d5fc4cc72bbc2eeef09ba3b9315f39b750624d3b2eeb1966b7a126960"}
Oct 31 03:14:26.528703 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:26.528353    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t" event={"ID":"e94e222c-4de2-4046-a659-0c92420cc03e","Type":"ContainerStarted","Data":"f60733e2b63440a02435ca09156dd54aec1c6a873ee6b7231c714a4f176a0bba"}
Oct 31 03:14:26.528703 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:26.528414    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t"
Oct 31 03:14:26.637556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:26.637491    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t" podStartSLOduration=1.777514927 podStartE2EDuration="2.637477335s" totalImagesPullingTime="859.962408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:26.636632988 +0000 UTC m=+408.197112718" watchObservedRunningTime="2026-10-31 03:14:26.637477335 +0000 UTC m=+408.197957064"
Oct 31 03:14:46.174530 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:14:46.174486    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:14:57.535441 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:57.535409    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cmm9t"
Oct 31 03:14:57.692639 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:57.692596    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"]
Oct 31 03:14:57.692896 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:57.692859    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s" podUID="180d2893-b9e6-4dc1-9bfd-00ca15cd286a" containerName="manager" containerID="cri-o://b2d6fc422fb685f06378aba7104a2ce7a7491af40944418f96d880b57c4e4282" gracePeriod=30
Oct 31 03:14:57.939935 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:57.939907    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"
Oct 31 03:14:57.954762 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:57.954725    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-cert\" (UniqueName: \"kubernetes.io/secret/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-cert\") pod \"180d2893-b9e6-4dc1-9bfd-00ca15cd286a\" (UID: \"180d2893-b9e6-4dc1-9bfd-00ca15cd286a\") "
Oct 31 03:14:57.954983 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:57.954820    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-kube-api-access-cf6p4\" (UniqueName: \"kubernetes.io/projected/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-kube-api-access-cf6p4\") pod \"180d2893-b9e6-4dc1-9bfd-00ca15cd286a\" (UID: \"180d2893-b9e6-4dc1-9bfd-00ca15cd286a\") "
Oct 31 03:14:57.957078 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:57.957047    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-cert" pod "180d2893-b9e6-4dc1-9bfd-00ca15cd286a" (UID: "180d2893-b9e6-4dc1-9bfd-00ca15cd286a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:14:57.957078 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:57.957059    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-kube-api-access-cf6p4" pod "180d2893-b9e6-4dc1-9bfd-00ca15cd286a" (UID: "180d2893-b9e6-4dc1-9bfd-00ca15cd286a"). InnerVolumeSpecName "kube-api-access-cf6p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:14:58.055865 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:58.055826    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf6p4\" (UniqueName: \"kubernetes.io/projected/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-kube-api-access-cf6p4\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:14:58.055865 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:58.055859    5415 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/180d2893-b9e6-4dc1-9bfd-00ca15cd286a-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:14:58.673274 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:58.673242    5415 generic.go:356] "Generic (PLEG): container finished" podID="180d2893-b9e6-4dc1-9bfd-00ca15cd286a" containerID="b2d6fc422fb685f06378aba7104a2ce7a7491af40944418f96d880b57c4e4282" exitCode=0
Oct 31 03:14:58.673665 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:58.673310    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"
Oct 31 03:14:58.673665 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:58.673325    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s" event={"ID":"180d2893-b9e6-4dc1-9bfd-00ca15cd286a","Type":"ContainerDied","Data":"b2d6fc422fb685f06378aba7104a2ce7a7491af40944418f96d880b57c4e4282"}
Oct 31 03:14:58.673665 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:58.673389    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s" event={"ID":"180d2893-b9e6-4dc1-9bfd-00ca15cd286a","Type":"ContainerDied","Data":"995eb9d37b394fe3f62a3774226dca2df57a2e2f9a26e10a13db161911f44826"}
Oct 31 03:14:58.673665 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:58.673420    5415 scope.go:122] "RemoveContainer" containerID="b2d6fc422fb685f06378aba7104a2ce7a7491af40944418f96d880b57c4e4282"
Oct 31 03:14:58.681853 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:58.681825    5415 scope.go:122] "RemoveContainer" containerID="b2d6fc422fb685f06378aba7104a2ce7a7491af40944418f96d880b57c4e4282"
Oct 31 03:14:58.682165 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:14:58.682146    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d6fc422fb685f06378aba7104a2ce7a7491af40944418f96d880b57c4e4282\": container with ID starting with b2d6fc422fb685f06378aba7104a2ce7a7491af40944418f96d880b57c4e4282 not found: ID does not exist" containerID="b2d6fc422fb685f06378aba7104a2ce7a7491af40944418f96d880b57c4e4282"
Oct 31 03:14:58.682242 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:58.682174    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d6fc422fb685f06378aba7104a2ce7a7491af40944418f96d880b57c4e4282"} err="failed to get container status \"b2d6fc422fb685f06378aba7104a2ce7a7491af40944418f96d880b57c4e4282\": rpc error: code = NotFound desc = could not find container \"b2d6fc422fb685f06378aba7104a2ce7a7491af40944418f96d880b57c4e4282\": container with ID starting with b2d6fc422fb685f06378aba7104a2ce7a7491af40944418f96d880b57c4e4282 not found: ID does not exist"
Oct 31 03:14:58.829737 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:58.829703    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"]
Oct 31 03:14:58.877047 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:14:58.876960    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sb49s"]
Oct 31 03:15:00.604640 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:15:00.604606    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="180d2893-b9e6-4dc1-9bfd-00ca15cd286a" path="/var/lib/kubelet/pods/180d2893-b9e6-4dc1-9bfd-00ca15cd286a/volumes"
Oct 31 03:15:16.174765 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:15:16.174681    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:15:46.174078 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:15:46.174038    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:16:16.174457 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:16:16.174417    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:16:20.255552 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.255509    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"]
Oct 31 03:16:20.256086 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.255964    5415 cpu_manager.go:414] "containerMap: removing container" podUID="180d2893-b9e6-4dc1-9bfd-00ca15cd286a" containerName="manager"
Oct 31 03:16:20.256086 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.255978    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180d2893-b9e6-4dc1-9bfd-00ca15cd286a" containerName="manager"
Oct 31 03:16:20.256086 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.256040    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="180d2893-b9e6-4dc1-9bfd-00ca15cd286a" containerName="manager"
Oct 31 03:16:20.258245 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.258226    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.307612 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.307584    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-96js4\""
Oct 31 03:16:20.354590 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.354551    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 03:16:20.356196 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.356170    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"]
Oct 31 03:16:20.408818 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.408779    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.408818 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.408818    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b20bf0-72db-4618-b347-c8db2b887b6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.409047 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.408918    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.409047 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.408948    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.409047 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.408977    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.409141 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.409108    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.409173 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.409141    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.409205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.409173    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.409237 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.409223    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prjsc\" (UniqueName: \"kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-kube-api-access-prjsc\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.509786 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.509689    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.509786 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.509750    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prjsc\" (UniqueName: \"kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-kube-api-access-prjsc\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.510044 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.509794    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.510044 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.509821    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b20bf0-72db-4618-b347-c8db2b887b6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.510044 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.509858    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.510044 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.509917    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.510044 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.509959    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.510044 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.510022    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.510338 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.510052    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.510338 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.510143    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.510549 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.510484    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.510756 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.510722    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.510891 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.510857    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b20bf0-72db-4618-b347-c8db2b887b6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.511010 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.510987    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.512667 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.512641    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.512908 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.512864    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.565201 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.565160    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.565201 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.565187    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prjsc\" (UniqueName: \"kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-kube-api-access-prjsc\") pod \"maas-default-gateway-openshift-default-6886b96f9-hbjsw\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.574050 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.574014    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:20.988752 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:20.988728    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"]
Oct 31 03:16:20.991432 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:16:20.991399    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b20bf0_72db_4618_b347_c8db2b887b6d.slice/crio-c9640d9642b729282dbd6e1158bfc865169c7c5b1101eaf1a9b20d30ebd02c43 WatchSource:0}: Error finding container c9640d9642b729282dbd6e1158bfc865169c7c5b1101eaf1a9b20d30ebd02c43: Status 404 returned error can't find the container with id c9640d9642b729282dbd6e1158bfc865169c7c5b1101eaf1a9b20d30ebd02c43
Oct 31 03:16:21.045829 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:21.045795    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" event={"ID":"c2b20bf0-72db-4618-b347-c8db2b887b6d","Type":"ContainerStarted","Data":"c9640d9642b729282dbd6e1158bfc865169c7c5b1101eaf1a9b20d30ebd02c43"}
Oct 31 03:16:25.860869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:25.860829    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:16:25.861162 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:25.860929    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:16:25.861162 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:25.860963    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:16:26.076033 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:26.075998    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" event={"ID":"c2b20bf0-72db-4618-b347-c8db2b887b6d","Type":"ContainerStarted","Data":"b6686b29bbde7b4f95f2591fa5a903eccda7d3eac431e346c37f03ed0d9f2fdc"}
Oct 31 03:16:26.235855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:26.235755    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" podStartSLOduration=1.3692116140000001 podStartE2EDuration="6.235740891s" totalImagesPullingTime="4.866529277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:26.234169788 +0000 UTC m=+527.794649537" watchObservedRunningTime="2026-10-31 03:16:26.235740891 +0000 UTC m=+527.796220620"
Oct 31 03:16:26.574917 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:26.574792    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:26.576395 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:26.576366    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hbjsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 03:16:26.576517 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:26.576462    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" podUID="c2b20bf0-72db-4618-b347-c8db2b887b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 03:16:27.574761 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:27.574723    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hbjsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 03:16:27.576963 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:27.574782    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" podUID="c2b20bf0-72db-4618-b347-c8db2b887b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 03:16:28.574666 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:28.574628    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hbjsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 03:16:28.574858 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:28.574686    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" podUID="c2b20bf0-72db-4618-b347-c8db2b887b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 03:16:29.574557 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:29.574518    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hbjsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 03:16:29.574771 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:29.574580    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" podUID="c2b20bf0-72db-4618-b347-c8db2b887b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 03:16:30.574594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:30.574557    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hbjsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 03:16:30.575002 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:30.574614    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" podUID="c2b20bf0-72db-4618-b347-c8db2b887b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 03:16:31.575222 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:31.575179    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hbjsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 03:16:31.575638 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:31.575239    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" podUID="c2b20bf0-72db-4618-b347-c8db2b887b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 03:16:32.575349 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:32.575311    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hbjsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 03:16:32.575846 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:32.575369    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" podUID="c2b20bf0-72db-4618-b347-c8db2b887b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 03:16:34.187819 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:34.187786    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:34.188358 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:34.188233    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:34.189067 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:34.189044    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:16:43.587522 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:43.587485    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d44779cff-9cj59"]
Oct 31 03:16:43.587988 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:43.587681    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-d44779cff-9cj59" podUID="037e56de-57bc-440c-a7d2-bb1f80aa0264" containerName="authorino" containerID="cri-o://8cb23fe66066fee4191b6c6c20f7188d0898760d152eb8702a5c53d306339a54" gracePeriod=30
Oct 31 03:16:43.836151 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:43.836124    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d44779cff-9cj59"
Oct 31 03:16:43.940163 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:43.940072    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/037e56de-57bc-440c-a7d2-bb1f80aa0264-tls-cert\" (UniqueName: \"kubernetes.io/secret/037e56de-57bc-440c-a7d2-bb1f80aa0264-tls-cert\") pod \"037e56de-57bc-440c-a7d2-bb1f80aa0264\" (UID: \"037e56de-57bc-440c-a7d2-bb1f80aa0264\") "
Oct 31 03:16:43.940329 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:43.940204    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037e56de-57bc-440c-a7d2-bb1f80aa0264-kube-api-access-zn4gr\" (UniqueName: \"kubernetes.io/projected/037e56de-57bc-440c-a7d2-bb1f80aa0264-kube-api-access-zn4gr\") pod \"037e56de-57bc-440c-a7d2-bb1f80aa0264\" (UID: \"037e56de-57bc-440c-a7d2-bb1f80aa0264\") "
Oct 31 03:16:43.942180 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:43.942145    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037e56de-57bc-440c-a7d2-bb1f80aa0264-kube-api-access-zn4gr" pod "037e56de-57bc-440c-a7d2-bb1f80aa0264" (UID: "037e56de-57bc-440c-a7d2-bb1f80aa0264"). InnerVolumeSpecName "kube-api-access-zn4gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:16:43.952016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:43.951963    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/037e56de-57bc-440c-a7d2-bb1f80aa0264-tls-cert" pod "037e56de-57bc-440c-a7d2-bb1f80aa0264" (UID: "037e56de-57bc-440c-a7d2-bb1f80aa0264"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:16:44.040955 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:44.040918    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn4gr\" (UniqueName: \"kubernetes.io/projected/037e56de-57bc-440c-a7d2-bb1f80aa0264-kube-api-access-zn4gr\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:44.040955 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:44.040950    5415 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/037e56de-57bc-440c-a7d2-bb1f80aa0264-tls-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:44.161564 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:44.161533    5415 generic.go:356] "Generic (PLEG): container finished" podID="037e56de-57bc-440c-a7d2-bb1f80aa0264" containerID="8cb23fe66066fee4191b6c6c20f7188d0898760d152eb8702a5c53d306339a54" exitCode=0
Oct 31 03:16:44.161745 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:44.161579    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d44779cff-9cj59"
Oct 31 03:16:44.161745 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:44.161606    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d44779cff-9cj59" event={"ID":"037e56de-57bc-440c-a7d2-bb1f80aa0264","Type":"ContainerDied","Data":"8cb23fe66066fee4191b6c6c20f7188d0898760d152eb8702a5c53d306339a54"}
Oct 31 03:16:44.161745 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:44.161644    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d44779cff-9cj59" event={"ID":"037e56de-57bc-440c-a7d2-bb1f80aa0264","Type":"ContainerDied","Data":"82696d41069207dd3f3e32585cbc2b92791ff8cfbd7baa817162df7ab9982921"}
Oct 31 03:16:44.161745 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:44.161661    5415 scope.go:122] "RemoveContainer" containerID="8cb23fe66066fee4191b6c6c20f7188d0898760d152eb8702a5c53d306339a54"
Oct 31 03:16:44.171390 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:44.171367    5415 scope.go:122] "RemoveContainer" containerID="8cb23fe66066fee4191b6c6c20f7188d0898760d152eb8702a5c53d306339a54"
Oct 31 03:16:44.171700 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:16:44.171678    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb23fe66066fee4191b6c6c20f7188d0898760d152eb8702a5c53d306339a54\": container with ID starting with 8cb23fe66066fee4191b6c6c20f7188d0898760d152eb8702a5c53d306339a54 not found: ID does not exist" containerID="8cb23fe66066fee4191b6c6c20f7188d0898760d152eb8702a5c53d306339a54"
Oct 31 03:16:44.171779 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:44.171710    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb23fe66066fee4191b6c6c20f7188d0898760d152eb8702a5c53d306339a54"} err="failed to get container status \"8cb23fe66066fee4191b6c6c20f7188d0898760d152eb8702a5c53d306339a54\": rpc error: code = NotFound desc = could not find container \"8cb23fe66066fee4191b6c6c20f7188d0898760d152eb8702a5c53d306339a54\": container with ID starting with 8cb23fe66066fee4191b6c6c20f7188d0898760d152eb8702a5c53d306339a54 not found: ID does not exist"
Oct 31 03:16:44.318751 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:44.318655    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d44779cff-9cj59"]
Oct 31 03:16:44.365647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:44.365609    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d44779cff-9cj59"]
Oct 31 03:16:44.603945 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:16:44.603913    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037e56de-57bc-440c-a7d2-bb1f80aa0264" path="/var/lib/kubelet/pods/037e56de-57bc-440c-a7d2-bb1f80aa0264/volumes"
Oct 31 03:16:46.173296 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:16:46.173219    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:17:16.176028 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:17:16.175982    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:17:31.291225 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.291190    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"]
Oct 31 03:17:31.291762 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.291585    5415 cpu_manager.go:414] "containerMap: removing container" podUID="037e56de-57bc-440c-a7d2-bb1f80aa0264" containerName="authorino"
Oct 31 03:17:31.291762 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.291596    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037e56de-57bc-440c-a7d2-bb1f80aa0264" containerName="authorino"
Oct 31 03:17:31.291762 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.291662    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="037e56de-57bc-440c-a7d2-bb1f80aa0264" containerName="authorino"
Oct 31 03:17:31.294638 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.294619    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"
Oct 31 03:17:31.344205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.344173    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4072c5d1\""
Oct 31 03:17:31.391460 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.391411    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4072c5d1-dockercfg-kmr7z\""
Oct 31 03:17:31.393097 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.393071    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"]
Oct 31 03:17:31.397845 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.397819    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbm56\" (UniqueName: \"kubernetes.io/projected/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-kube-api-access-gbm56\") pod \"payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml\" (UID: \"fffc0ea6-4fa4-4bad-81ed-b9a29d014848\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"
Oct 31 03:17:31.397939 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.397854    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml\" (UID: \"fffc0ea6-4fa4-4bad-81ed-b9a29d014848\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"
Oct 31 03:17:31.498778 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.498738    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbm56\" (UniqueName: \"kubernetes.io/projected/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-kube-api-access-gbm56\") pod \"payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml\" (UID: \"fffc0ea6-4fa4-4bad-81ed-b9a29d014848\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"
Oct 31 03:17:31.498778 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.498779    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml\" (UID: \"fffc0ea6-4fa4-4bad-81ed-b9a29d014848\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"
Oct 31 03:17:31.499490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.499468    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml\" (UID: \"fffc0ea6-4fa4-4bad-81ed-b9a29d014848\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"
Oct 31 03:17:31.553834 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.553766    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbm56\" (UniqueName: \"kubernetes.io/projected/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-kube-api-access-gbm56\") pod \"payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml\" (UID: \"fffc0ea6-4fa4-4bad-81ed-b9a29d014848\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"
Oct 31 03:17:31.607997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.607954    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"
Oct 31 03:17:31.981739 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:31.981680    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"]
Oct 31 03:17:31.984633 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:17:31.984602    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffc0ea6_4fa4_4bad_81ed_b9a29d014848.slice/crio-bc50dfa6bdec6d1b1e0935151e322a910d7bb490af093cd5613f7e3a9f9dea67 WatchSource:0}: Error finding container bc50dfa6bdec6d1b1e0935151e322a910d7bb490af093cd5613f7e3a9f9dea67: Status 404 returned error can't find the container with id bc50dfa6bdec6d1b1e0935151e322a910d7bb490af093cd5613f7e3a9f9dea67
Oct 31 03:17:32.378291 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:32.378249    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml" event={"ID":"fffc0ea6-4fa4-4bad-81ed-b9a29d014848","Type":"ContainerStarted","Data":"bc50dfa6bdec6d1b1e0935151e322a910d7bb490af093cd5613f7e3a9f9dea67"}
Oct 31 03:17:33.698853 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.698814    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"]
Oct 31 03:17:33.703163 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.703132    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"
Oct 31 03:17:33.752596 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.752545    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-46d24f95-dockercfg-cwjp5\""
Oct 31 03:17:33.800036 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.799986    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-46d24f95\""
Oct 31 03:17:33.801573 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.801542    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"]
Oct 31 03:17:33.820487 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.820453    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f718c5-2831-4d12-acee-b4adb4dd8735-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq\" (UID: \"46f718c5-2831-4d12-acee-b4adb4dd8735\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"
Oct 31 03:17:33.820672 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.820536    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxb9\" (UniqueName: \"kubernetes.io/projected/46f718c5-2831-4d12-acee-b4adb4dd8735-kube-api-access-xhxb9\") pod \"payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq\" (UID: \"46f718c5-2831-4d12-acee-b4adb4dd8735\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"
Oct 31 03:17:33.861475 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.861439    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"]
Oct 31 03:17:33.864885 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.864850    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"
Oct 31 03:17:33.921639 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.921597    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f718c5-2831-4d12-acee-b4adb4dd8735-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq\" (UID: \"46f718c5-2831-4d12-acee-b4adb4dd8735\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"
Oct 31 03:17:33.921639 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.921650    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ngfp\" (UniqueName: \"kubernetes.io/projected/877ceae3-b42e-4783-8ddc-5b1d56fde794-kube-api-access-8ngfp\") pod \"payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5\" (UID: \"877ceae3-b42e-4783-8ddc-5b1d56fde794\") " pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"
Oct 31 03:17:33.921947 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.921713    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877ceae3-b42e-4783-8ddc-5b1d56fde794-plugins-config-volume\") pod \"payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5\" (UID: \"877ceae3-b42e-4783-8ddc-5b1d56fde794\") " pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"
Oct 31 03:17:33.921947 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.921746    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxb9\" (UniqueName: \"kubernetes.io/projected/46f718c5-2831-4d12-acee-b4adb4dd8735-kube-api-access-xhxb9\") pod \"payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq\" (UID: \"46f718c5-2831-4d12-acee-b4adb4dd8735\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"
Oct 31 03:17:33.922860 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.922836    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f718c5-2831-4d12-acee-b4adb4dd8735-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq\" (UID: \"46f718c5-2831-4d12-acee-b4adb4dd8735\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"
Oct 31 03:17:33.923385 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.923362    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"]
Oct 31 03:17:33.977486 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:33.977416    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxb9\" (UniqueName: \"kubernetes.io/projected/46f718c5-2831-4d12-acee-b4adb4dd8735-kube-api-access-xhxb9\") pod \"payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq\" (UID: \"46f718c5-2831-4d12-acee-b4adb4dd8735\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"
Oct 31 03:17:34.020518 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:34.020479    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"
Oct 31 03:17:34.022442 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:34.022414    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877ceae3-b42e-4783-8ddc-5b1d56fde794-plugins-config-volume\") pod \"payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5\" (UID: \"877ceae3-b42e-4783-8ddc-5b1d56fde794\") " pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"
Oct 31 03:17:34.022593 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:34.022503    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ngfp\" (UniqueName: \"kubernetes.io/projected/877ceae3-b42e-4783-8ddc-5b1d56fde794-kube-api-access-8ngfp\") pod \"payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5\" (UID: \"877ceae3-b42e-4783-8ddc-5b1d56fde794\") " pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"
Oct 31 03:17:34.023262 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:34.023232    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877ceae3-b42e-4783-8ddc-5b1d56fde794-plugins-config-volume\") pod \"payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5\" (UID: \"877ceae3-b42e-4783-8ddc-5b1d56fde794\") " pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"
Oct 31 03:17:34.077259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:34.077224    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ngfp\" (UniqueName: \"kubernetes.io/projected/877ceae3-b42e-4783-8ddc-5b1d56fde794-kube-api-access-8ngfp\") pod \"payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5\" (UID: \"877ceae3-b42e-4783-8ddc-5b1d56fde794\") " pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"
Oct 31 03:17:34.182779 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:34.182741    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"
Oct 31 03:17:34.411136 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:34.411103    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"]
Oct 31 03:17:34.414702 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:17:34.414666    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f718c5_2831_4d12_acee_b4adb4dd8735.slice/crio-4aa5d610f684b85dcf89df4d5e06f7d00e90e15ab1dc4db163072b96764b8f3b WatchSource:0}: Error finding container 4aa5d610f684b85dcf89df4d5e06f7d00e90e15ab1dc4db163072b96764b8f3b: Status 404 returned error can't find the container with id 4aa5d610f684b85dcf89df4d5e06f7d00e90e15ab1dc4db163072b96764b8f3b
Oct 31 03:17:34.563418 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:34.563384    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"]
Oct 31 03:17:34.566414 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:17:34.566383    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877ceae3_b42e_4783_8ddc_5b1d56fde794.slice/crio-0f991217dc91753b0d3b89437e882482e165fe3bd6ef8a493c1e554e3f64ec47 WatchSource:0}: Error finding container 0f991217dc91753b0d3b89437e882482e165fe3bd6ef8a493c1e554e3f64ec47: Status 404 returned error can't find the container with id 0f991217dc91753b0d3b89437e882482e165fe3bd6ef8a493c1e554e3f64ec47
Oct 31 03:17:35.394614 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:35.394573    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq" event={"ID":"46f718c5-2831-4d12-acee-b4adb4dd8735","Type":"ContainerStarted","Data":"58dba905a29c8015838e895efe13303b7e2945fdd05681df1eeda795a7ba1bff"}
Oct 31 03:17:35.395119 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:35.394637    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"
Oct 31 03:17:35.395119 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:35.394658    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq" event={"ID":"46f718c5-2831-4d12-acee-b4adb4dd8735","Type":"ContainerStarted","Data":"4aa5d610f684b85dcf89df4d5e06f7d00e90e15ab1dc4db163072b96764b8f3b"}
Oct 31 03:17:35.396146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:35.396123    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml" event={"ID":"fffc0ea6-4fa4-4bad-81ed-b9a29d014848","Type":"ContainerStarted","Data":"d797a9d34246e2ec2ef8ebef3f10c3234749cbb1eb0bf5c14bce4254417ec310"}
Oct 31 03:17:35.396253 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:35.396211    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"
Oct 31 03:17:35.397344 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:35.397323    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5" event={"ID":"877ceae3-b42e-4783-8ddc-5b1d56fde794","Type":"ContainerStarted","Data":"0f991217dc91753b0d3b89437e882482e165fe3bd6ef8a493c1e554e3f64ec47"}
Oct 31 03:17:35.505988 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:35.505936    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq" podStartSLOduration=1.686552093 podStartE2EDuration="2.505920564s" totalImagesPullingTime="819.368471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:35.505175274 +0000 UTC m=+597.065655030" watchObservedRunningTime="2026-10-31 03:17:35.505920564 +0000 UTC m=+597.066400295"
Oct 31 03:17:35.615810 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:35.615760    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml" podStartSLOduration=1.785630777 podStartE2EDuration="4.615745049s" totalImagesPullingTime="2.830114272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:35.614374004 +0000 UTC m=+597.174853735" watchObservedRunningTime="2026-10-31 03:17:35.615745049 +0000 UTC m=+597.176224780"
Oct 31 03:17:36.403385 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:36.403344    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5" event={"ID":"877ceae3-b42e-4783-8ddc-5b1d56fde794","Type":"ContainerStarted","Data":"83666ae87dd3d6b29e46ae061fdc8f766cfe4e8f82827ae27debc477c8ded763"}
Oct 31 03:17:36.403861 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:36.403487    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"
Oct 31 03:17:36.514216 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:36.514156    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5" podStartSLOduration=2.646414925 podStartE2EDuration="3.514139835s" totalImagesPullingTime="867.72491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:36.512723378 +0000 UTC m=+598.073203128" watchObservedRunningTime="2026-10-31 03:17:36.514139835 +0000 UTC m=+598.074619633"
Oct 31 03:17:46.175095 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:17:46.175049    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:17:46.406749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:46.406716    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"
Oct 31 03:17:46.406969 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:46.406803    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"
Oct 31 03:17:47.420817 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:47.420786    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"
Oct 31 03:17:52.840064 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:52.840027    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"]
Oct 31 03:17:52.843530 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:52.843509    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:52.899792 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:52.899749    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-maas-api-tls\") pod \"maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:52.900016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:52.899802    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-metrics-certs\") pod \"maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:52.900016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:52.899832    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfz84\" (UniqueName: \"kubernetes.io/projected/21e370b8-a051-41f5-9641-9cd425369e48-kube-api-access-nfz84\") pod \"maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:52.900016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:52.899991    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e370b8-a051-41f5-9641-9cd425369e48-openshift-service-ca\") pod \"maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:52.940301 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:52.940268    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-46d24f95-serving-cert\""
Oct 31 03:17:52.940632 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:52.940608    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 03:17:52.940732 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:52.940717    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jsn62\""
Oct 31 03:17:52.942008 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:52.941990    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"]
Oct 31 03:17:53.000349 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:53.000318    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-metrics-certs\") pod \"maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:53.000349 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:53.000353    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfz84\" (UniqueName: \"kubernetes.io/projected/21e370b8-a051-41f5-9641-9cd425369e48-kube-api-access-nfz84\") pod \"maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:53.000584 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:53.000446    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e370b8-a051-41f5-9641-9cd425369e48-openshift-service-ca\") pod \"maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:53.000644 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:53.000594    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-maas-api-tls\") pod \"maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:53.000766 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:17:53.000744    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-serving-cert: secret "maas-api-e2e-ait-46d24f95-serving-cert" not found
Oct 31 03:17:53.000837 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:17:53.000826    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-maas-api-tls podName:21e370b8-a051-41f5-9641-9cd425369e48 nodeName:}" failed. No retries permitted until 2026-10-31 03:17:53.50080823 +0000 UTC m=+615.061287954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-maas-api-tls") pod "maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg" (UID: "21e370b8-a051-41f5-9641-9cd425369e48") : secret "maas-api-e2e-ait-46d24f95-serving-cert" not found
Oct 31 03:17:53.001188 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:53.001169    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e370b8-a051-41f5-9641-9cd425369e48-openshift-service-ca\") pod \"maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:53.003003 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:53.002980    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-metrics-certs\") pod \"maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:53.056025 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:53.055995    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfz84\" (UniqueName: \"kubernetes.io/projected/21e370b8-a051-41f5-9641-9cd425369e48-kube-api-access-nfz84\") pod \"maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:53.506010 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:53.505971    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-maas-api-tls\") pod \"maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:53.508413 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:53.508384    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-maas-api-tls\") pod \"maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:53.756576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:53.756478    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:54.123841 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:54.123817    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"]
Oct 31 03:17:54.126403 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:17:54.126373    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e370b8_a051_41f5_9641_9cd425369e48.slice/crio-95a62f563070649989523ded57958f7d501b20047c4193a6dfe2aa6bafb01618 WatchSource:0}: Error finding container 95a62f563070649989523ded57958f7d501b20047c4193a6dfe2aa6bafb01618: Status 404 returned error can't find the container with id 95a62f563070649989523ded57958f7d501b20047c4193a6dfe2aa6bafb01618
Oct 31 03:17:54.128091 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:54.128071    5415 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:17:54.495536 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:54.495448    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg" event={"ID":"21e370b8-a051-41f5-9641-9cd425369e48","Type":"ContainerStarted","Data":"95a62f563070649989523ded57958f7d501b20047c4193a6dfe2aa6bafb01618"}
Oct 31 03:17:57.512594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:57.512497    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg" event={"ID":"21e370b8-a051-41f5-9641-9cd425369e48","Type":"ContainerStarted","Data":"2cc7261dd0a7e1b34fb8554bf922bb42bae1853013256cbe9d3a78d9fce6c0c1"}
Oct 31 03:17:57.513003 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:57.512628    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:17:57.623187 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:17:57.623133    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg" podStartSLOduration=2.585188202 podStartE2EDuration="5.623118886s" totalImagesPullingTime="3.037930684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:57.621999276 +0000 UTC m=+619.182479007" watchObservedRunningTime="2026-10-31 03:17:57.623118886 +0000 UTC m=+619.183598616"
Oct 31 03:18:03.523209 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:03.523176    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:18:16.176000 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:18:16.175908    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:18:18.287919 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:18.287863    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"]
Oct 31 03:18:23.288416 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.288344    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml" podUID="fffc0ea6-4fa4-4bad-81ed-b9a29d014848" containerName="payload-pre-processing" containerID="cri-o://d797a9d34246e2ec2ef8ebef3f10c3234749cbb1eb0bf5c14bce4254417ec310" gracePeriod=25
Oct 31 03:18:23.544191 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.544129    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"
Oct 31 03:18:23.632396 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.632362    5415 generic.go:356] "Generic (PLEG): container finished" podID="fffc0ea6-4fa4-4bad-81ed-b9a29d014848" containerID="d797a9d34246e2ec2ef8ebef3f10c3234749cbb1eb0bf5c14bce4254417ec310" exitCode=0
Oct 31 03:18:23.632570 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.632415    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"
Oct 31 03:18:23.632570 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.632456    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml" event={"ID":"fffc0ea6-4fa4-4bad-81ed-b9a29d014848","Type":"ContainerDied","Data":"d797a9d34246e2ec2ef8ebef3f10c3234749cbb1eb0bf5c14bce4254417ec310"}
Oct 31 03:18:23.632570 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.632510    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml" event={"ID":"fffc0ea6-4fa4-4bad-81ed-b9a29d014848","Type":"ContainerDied","Data":"bc50dfa6bdec6d1b1e0935151e322a910d7bb490af093cd5613f7e3a9f9dea67"}
Oct 31 03:18:23.632570 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.632535    5415 scope.go:122] "RemoveContainer" containerID="d797a9d34246e2ec2ef8ebef3f10c3234749cbb1eb0bf5c14bce4254417ec310"
Oct 31 03:18:23.643178 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.643151    5415 scope.go:122] "RemoveContainer" containerID="d797a9d34246e2ec2ef8ebef3f10c3234749cbb1eb0bf5c14bce4254417ec310"
Oct 31 03:18:23.643501 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:18:23.643482    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d797a9d34246e2ec2ef8ebef3f10c3234749cbb1eb0bf5c14bce4254417ec310\": container with ID starting with d797a9d34246e2ec2ef8ebef3f10c3234749cbb1eb0bf5c14bce4254417ec310 not found: ID does not exist" containerID="d797a9d34246e2ec2ef8ebef3f10c3234749cbb1eb0bf5c14bce4254417ec310"
Oct 31 03:18:23.643561 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.643516    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d797a9d34246e2ec2ef8ebef3f10c3234749cbb1eb0bf5c14bce4254417ec310"} err="failed to get container status \"d797a9d34246e2ec2ef8ebef3f10c3234749cbb1eb0bf5c14bce4254417ec310\": rpc error: code = NotFound desc = could not find container \"d797a9d34246e2ec2ef8ebef3f10c3234749cbb1eb0bf5c14bce4254417ec310\": container with ID starting with d797a9d34246e2ec2ef8ebef3f10c3234749cbb1eb0bf5c14bce4254417ec310 not found: ID does not exist"
Oct 31 03:18:23.679720 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.679685    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-plugins-config-volume\") pod \"fffc0ea6-4fa4-4bad-81ed-b9a29d014848\" (UID: \"fffc0ea6-4fa4-4bad-81ed-b9a29d014848\") "
Oct 31 03:18:23.679930 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.679730    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-kube-api-access-gbm56\" (UniqueName: \"kubernetes.io/projected/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-kube-api-access-gbm56\") pod \"fffc0ea6-4fa4-4bad-81ed-b9a29d014848\" (UID: \"fffc0ea6-4fa4-4bad-81ed-b9a29d014848\") "
Oct 31 03:18:23.680179 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.680149    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-plugins-config-volume" pod "fffc0ea6-4fa4-4bad-81ed-b9a29d014848" (UID: "fffc0ea6-4fa4-4bad-81ed-b9a29d014848"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:23.682178 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.682147    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-kube-api-access-gbm56" pod "fffc0ea6-4fa4-4bad-81ed-b9a29d014848" (UID: "fffc0ea6-4fa4-4bad-81ed-b9a29d014848"). InnerVolumeSpecName "kube-api-access-gbm56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:23.781141 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.781101    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:23.781141 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:23.781135    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbm56\" (UniqueName: \"kubernetes.io/projected/fffc0ea6-4fa4-4bad-81ed-b9a29d014848-kube-api-access-gbm56\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:24.046004 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:24.045964    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"]
Oct 31 03:18:24.093153 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:24.093117    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4072c5d1-6c6b757f8c-s9xml"]
Oct 31 03:18:24.604744 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:24.604696    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fffc0ea6-4fa4-4bad-81ed-b9a29d014848" path="/var/lib/kubelet/pods/fffc0ea6-4fa4-4bad-81ed-b9a29d014848/volumes"
Oct 31 03:18:46.175233 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:18:46.175193    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:18:48.479128 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:48.479088    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"]
Oct 31 03:18:48.588995 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:48.588960    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"]
Oct 31 03:18:53.479757 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.479691    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5" podUID="877ceae3-b42e-4783-8ddc-5b1d56fde794" containerName="payload-processing" containerID="cri-o://83666ae87dd3d6b29e46ae061fdc8f766cfe4e8f82827ae27debc477c8ded763" gracePeriod=25
Oct 31 03:18:53.589326 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.589258    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq" podUID="46f718c5-2831-4d12-acee-b4adb4dd8735" containerName="payload-pre-processing" containerID="cri-o://58dba905a29c8015838e895efe13303b7e2945fdd05681df1eeda795a7ba1bff" gracePeriod=25
Oct 31 03:18:53.752828 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.752800    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"
Oct 31 03:18:53.778834 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.778796    5415 generic.go:356] "Generic (PLEG): container finished" podID="877ceae3-b42e-4783-8ddc-5b1d56fde794" containerID="83666ae87dd3d6b29e46ae061fdc8f766cfe4e8f82827ae27debc477c8ded763" exitCode=0
Oct 31 03:18:53.779027 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.778838    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"
Oct 31 03:18:53.779027 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.778872    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5" event={"ID":"877ceae3-b42e-4783-8ddc-5b1d56fde794","Type":"ContainerDied","Data":"83666ae87dd3d6b29e46ae061fdc8f766cfe4e8f82827ae27debc477c8ded763"}
Oct 31 03:18:53.779027 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.778961    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5" event={"ID":"877ceae3-b42e-4783-8ddc-5b1d56fde794","Type":"ContainerDied","Data":"0f991217dc91753b0d3b89437e882482e165fe3bd6ef8a493c1e554e3f64ec47"}
Oct 31 03:18:53.779027 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.778990    5415 scope.go:122] "RemoveContainer" containerID="83666ae87dd3d6b29e46ae061fdc8f766cfe4e8f82827ae27debc477c8ded763"
Oct 31 03:18:53.780813 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.780793    5415 generic.go:356] "Generic (PLEG): container finished" podID="46f718c5-2831-4d12-acee-b4adb4dd8735" containerID="58dba905a29c8015838e895efe13303b7e2945fdd05681df1eeda795a7ba1bff" exitCode=0
Oct 31 03:18:53.780948 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.780836    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq" event={"ID":"46f718c5-2831-4d12-acee-b4adb4dd8735","Type":"ContainerDied","Data":"58dba905a29c8015838e895efe13303b7e2945fdd05681df1eeda795a7ba1bff"}
Oct 31 03:18:53.789043 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.789016    5415 scope.go:122] "RemoveContainer" containerID="83666ae87dd3d6b29e46ae061fdc8f766cfe4e8f82827ae27debc477c8ded763"
Oct 31 03:18:53.789420 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:18:53.789397    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83666ae87dd3d6b29e46ae061fdc8f766cfe4e8f82827ae27debc477c8ded763\": container with ID starting with 83666ae87dd3d6b29e46ae061fdc8f766cfe4e8f82827ae27debc477c8ded763 not found: ID does not exist" containerID="83666ae87dd3d6b29e46ae061fdc8f766cfe4e8f82827ae27debc477c8ded763"
Oct 31 03:18:53.789527 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.789433    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83666ae87dd3d6b29e46ae061fdc8f766cfe4e8f82827ae27debc477c8ded763"} err="failed to get container status \"83666ae87dd3d6b29e46ae061fdc8f766cfe4e8f82827ae27debc477c8ded763\": rpc error: code = NotFound desc = could not find container \"83666ae87dd3d6b29e46ae061fdc8f766cfe4e8f82827ae27debc477c8ded763\": container with ID starting with 83666ae87dd3d6b29e46ae061fdc8f766cfe4e8f82827ae27debc477c8ded763 not found: ID does not exist"
Oct 31 03:18:53.873667 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.873633    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"
Oct 31 03:18:53.874348 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.874315    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877ceae3-b42e-4783-8ddc-5b1d56fde794-kube-api-access-8ngfp\" (UniqueName: \"kubernetes.io/projected/877ceae3-b42e-4783-8ddc-5b1d56fde794-kube-api-access-8ngfp\") pod \"877ceae3-b42e-4783-8ddc-5b1d56fde794\" (UID: \"877ceae3-b42e-4783-8ddc-5b1d56fde794\") "
Oct 31 03:18:53.874520 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.874395    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/877ceae3-b42e-4783-8ddc-5b1d56fde794-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877ceae3-b42e-4783-8ddc-5b1d56fde794-plugins-config-volume\") pod \"877ceae3-b42e-4783-8ddc-5b1d56fde794\" (UID: \"877ceae3-b42e-4783-8ddc-5b1d56fde794\") "
Oct 31 03:18:53.875126 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.875097    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877ceae3-b42e-4783-8ddc-5b1d56fde794-plugins-config-volume" pod "877ceae3-b42e-4783-8ddc-5b1d56fde794" (UID: "877ceae3-b42e-4783-8ddc-5b1d56fde794"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:53.876605 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.876581    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877ceae3-b42e-4783-8ddc-5b1d56fde794-kube-api-access-8ngfp" pod "877ceae3-b42e-4783-8ddc-5b1d56fde794" (UID: "877ceae3-b42e-4783-8ddc-5b1d56fde794"). InnerVolumeSpecName "kube-api-access-8ngfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:53.975142 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.975105    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f718c5-2831-4d12-acee-b4adb4dd8735-kube-api-access-xhxb9\" (UniqueName: \"kubernetes.io/projected/46f718c5-2831-4d12-acee-b4adb4dd8735-kube-api-access-xhxb9\") pod \"46f718c5-2831-4d12-acee-b4adb4dd8735\" (UID: \"46f718c5-2831-4d12-acee-b4adb4dd8735\") "
Oct 31 03:18:53.975353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.975153    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46f718c5-2831-4d12-acee-b4adb4dd8735-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f718c5-2831-4d12-acee-b4adb4dd8735-plugins-config-volume\") pod \"46f718c5-2831-4d12-acee-b4adb4dd8735\" (UID: \"46f718c5-2831-4d12-acee-b4adb4dd8735\") "
Oct 31 03:18:53.975423 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.975370    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ngfp\" (UniqueName: \"kubernetes.io/projected/877ceae3-b42e-4783-8ddc-5b1d56fde794-kube-api-access-8ngfp\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:53.975423 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.975381    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877ceae3-b42e-4783-8ddc-5b1d56fde794-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:53.975556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.975531    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f718c5-2831-4d12-acee-b4adb4dd8735-plugins-config-volume" pod "46f718c5-2831-4d12-acee-b4adb4dd8735" (UID: "46f718c5-2831-4d12-acee-b4adb4dd8735"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:53.977340 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:53.977318    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f718c5-2831-4d12-acee-b4adb4dd8735-kube-api-access-xhxb9" pod "46f718c5-2831-4d12-acee-b4adb4dd8735" (UID: "46f718c5-2831-4d12-acee-b4adb4dd8735"). InnerVolumeSpecName "kube-api-access-xhxb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:54.076359 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:54.076272    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhxb9\" (UniqueName: \"kubernetes.io/projected/46f718c5-2831-4d12-acee-b4adb4dd8735-kube-api-access-xhxb9\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:54.076359 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:54.076302    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f718c5-2831-4d12-acee-b4adb4dd8735-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:54.192523 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:54.192485    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"]
Oct 31 03:18:54.195560 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:54.195533    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46d24f95-6698fc8f87-glxd5"]
Oct 31 03:18:54.604923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:54.604892    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="877ceae3-b42e-4783-8ddc-5b1d56fde794" path="/var/lib/kubelet/pods/877ceae3-b42e-4783-8ddc-5b1d56fde794/volumes"
Oct 31 03:18:54.787383 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:54.787294    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"
Oct 31 03:18:54.787548 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:54.787292    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq" event={"ID":"46f718c5-2831-4d12-acee-b4adb4dd8735","Type":"ContainerDied","Data":"4aa5d610f684b85dcf89df4d5e06f7d00e90e15ab1dc4db163072b96764b8f3b"}
Oct 31 03:18:54.787548 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:54.787478    5415 scope.go:122] "RemoveContainer" containerID="58dba905a29c8015838e895efe13303b7e2945fdd05681df1eeda795a7ba1bff"
Oct 31 03:18:54.950641 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:54.950604    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"]
Oct 31 03:18:54.955627 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:54.955597    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46d24f95-657bfdc4c5-mbwkq"]
Oct 31 03:18:56.604615 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:18:56.604585    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46f718c5-2831-4d12-acee-b4adb4dd8735" path="/var/lib/kubelet/pods/46f718c5-2831-4d12-acee-b4adb4dd8735/volumes"
Oct 31 03:19:01.882296 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:01.882260    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"]
Oct 31 03:19:01.882725 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:01.882458    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg" podUID="21e370b8-a051-41f5-9641-9cd425369e48" containerName="maas-api" containerID="cri-o://2cc7261dd0a7e1b34fb8554bf922bb42bae1853013256cbe9d3a78d9fce6c0c1" gracePeriod=30
Oct 31 03:19:02.151111 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.151086    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:19:02.253334 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.253300    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-maas-api-tls\") pod \"21e370b8-a051-41f5-9641-9cd425369e48\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") "
Oct 31 03:19:02.253334 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.253352    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e370b8-a051-41f5-9641-9cd425369e48-kube-api-access-nfz84\" (UniqueName: \"kubernetes.io/projected/21e370b8-a051-41f5-9641-9cd425369e48-kube-api-access-nfz84\") pod \"21e370b8-a051-41f5-9641-9cd425369e48\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") "
Oct 31 03:19:02.253606 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.253395    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-metrics-certs\") pod \"21e370b8-a051-41f5-9641-9cd425369e48\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") "
Oct 31 03:19:02.253606 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.253429    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e370b8-a051-41f5-9641-9cd425369e48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e370b8-a051-41f5-9641-9cd425369e48-openshift-service-ca\") pod \"21e370b8-a051-41f5-9641-9cd425369e48\" (UID: \"21e370b8-a051-41f5-9641-9cd425369e48\") "
Oct 31 03:19:02.253868 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.253843    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e370b8-a051-41f5-9641-9cd425369e48-openshift-service-ca" pod "21e370b8-a051-41f5-9641-9cd425369e48" (UID: "21e370b8-a051-41f5-9641-9cd425369e48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:19:02.255606 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.255582    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-maas-api-tls" pod "21e370b8-a051-41f5-9641-9cd425369e48" (UID: "21e370b8-a051-41f5-9641-9cd425369e48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:19:02.256013 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.255995    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e370b8-a051-41f5-9641-9cd425369e48-kube-api-access-nfz84" pod "21e370b8-a051-41f5-9641-9cd425369e48" (UID: "21e370b8-a051-41f5-9641-9cd425369e48"). InnerVolumeSpecName "kube-api-access-nfz84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:19:02.256100 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.255993    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-metrics-certs" pod "21e370b8-a051-41f5-9641-9cd425369e48" (UID: "21e370b8-a051-41f5-9641-9cd425369e48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:19:02.354941 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.354904    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:19:02.354941 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.354936    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfz84\" (UniqueName: \"kubernetes.io/projected/21e370b8-a051-41f5-9641-9cd425369e48-kube-api-access-nfz84\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:19:02.354941 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.354946    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e370b8-a051-41f5-9641-9cd425369e48-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:19:02.355190 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.354956    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e370b8-a051-41f5-9641-9cd425369e48-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:19:02.827862 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.827828    5415 generic.go:356] "Generic (PLEG): container finished" podID="21e370b8-a051-41f5-9641-9cd425369e48" containerID="2cc7261dd0a7e1b34fb8554bf922bb42bae1853013256cbe9d3a78d9fce6c0c1" exitCode=0
Oct 31 03:19:02.828035 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.827889    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"
Oct 31 03:19:02.828035 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.827916    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg" event={"ID":"21e370b8-a051-41f5-9641-9cd425369e48","Type":"ContainerDied","Data":"2cc7261dd0a7e1b34fb8554bf922bb42bae1853013256cbe9d3a78d9fce6c0c1"}
Oct 31 03:19:02.828035 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.827968    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg" event={"ID":"21e370b8-a051-41f5-9641-9cd425369e48","Type":"ContainerDied","Data":"95a62f563070649989523ded57958f7d501b20047c4193a6dfe2aa6bafb01618"}
Oct 31 03:19:02.828035 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.827990    5415 scope.go:122] "RemoveContainer" containerID="2cc7261dd0a7e1b34fb8554bf922bb42bae1853013256cbe9d3a78d9fce6c0c1"
Oct 31 03:19:02.836720 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.836698    5415 scope.go:122] "RemoveContainer" containerID="2cc7261dd0a7e1b34fb8554bf922bb42bae1853013256cbe9d3a78d9fce6c0c1"
Oct 31 03:19:02.837009 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:19:02.836984    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc7261dd0a7e1b34fb8554bf922bb42bae1853013256cbe9d3a78d9fce6c0c1\": container with ID starting with 2cc7261dd0a7e1b34fb8554bf922bb42bae1853013256cbe9d3a78d9fce6c0c1 not found: ID does not exist" containerID="2cc7261dd0a7e1b34fb8554bf922bb42bae1853013256cbe9d3a78d9fce6c0c1"
Oct 31 03:19:02.837077 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.837022    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc7261dd0a7e1b34fb8554bf922bb42bae1853013256cbe9d3a78d9fce6c0c1"} err="failed to get container status \"2cc7261dd0a7e1b34fb8554bf922bb42bae1853013256cbe9d3a78d9fce6c0c1\": rpc error: code = NotFound desc = could not find container \"2cc7261dd0a7e1b34fb8554bf922bb42bae1853013256cbe9d3a78d9fce6c0c1\": container with ID starting with 2cc7261dd0a7e1b34fb8554bf922bb42bae1853013256cbe9d3a78d9fce6c0c1 not found: ID does not exist"
Oct 31 03:19:02.982462 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:02.982381    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"]
Oct 31 03:19:03.029569 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.029531    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46d24f95-7884ffbb67-wnvfg"]
Oct 31 03:19:03.655320 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.655289    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"]
Oct 31 03:19:03.655681 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.655664    5415 cpu_manager.go:414] "containerMap: removing container" podUID="fffc0ea6-4fa4-4bad-81ed-b9a29d014848" containerName="payload-pre-processing"
Oct 31 03:19:03.655681 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.655678    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffc0ea6-4fa4-4bad-81ed-b9a29d014848" containerName="payload-pre-processing"
Oct 31 03:19:03.655833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.655697    5415 cpu_manager.go:414] "containerMap: removing container" podUID="46f718c5-2831-4d12-acee-b4adb4dd8735" containerName="payload-pre-processing"
Oct 31 03:19:03.655833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.655702    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f718c5-2831-4d12-acee-b4adb4dd8735" containerName="payload-pre-processing"
Oct 31 03:19:03.655833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.655716    5415 cpu_manager.go:414] "containerMap: removing container" podUID="21e370b8-a051-41f5-9641-9cd425369e48" containerName="maas-api"
Oct 31 03:19:03.655833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.655721    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e370b8-a051-41f5-9641-9cd425369e48" containerName="maas-api"
Oct 31 03:19:03.655833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.655733    5415 cpu_manager.go:414] "containerMap: removing container" podUID="877ceae3-b42e-4783-8ddc-5b1d56fde794" containerName="payload-processing"
Oct 31 03:19:03.655833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.655738    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877ceae3-b42e-4783-8ddc-5b1d56fde794" containerName="payload-processing"
Oct 31 03:19:03.655833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.655800    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f718c5-2831-4d12-acee-b4adb4dd8735" containerName="payload-pre-processing"
Oct 31 03:19:03.655833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.655809    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e370b8-a051-41f5-9641-9cd425369e48" containerName="maas-api"
Oct 31 03:19:03.655833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.655816    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="877ceae3-b42e-4783-8ddc-5b1d56fde794" containerName="payload-processing"
Oct 31 03:19:03.655833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.655826    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffc0ea6-4fa4-4bad-81ed-b9a29d014848" containerName="payload-pre-processing"
Oct 31 03:19:03.660062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.660043    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:03.757993 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.757951    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-9360eb-serving-cert\""
Oct 31 03:19:03.757993 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.757973    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jsn62\""
Oct 31 03:19:03.758272 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.758067    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 03:19:03.759993 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.759968    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"]
Oct 31 03:19:03.767613 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.767588    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:03.767763 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.767619    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:03.767763 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.767638    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkvlg\" (UniqueName: \"kubernetes.io/projected/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-kube-api-access-bkvlg\") pod \"maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:03.767763 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.767716    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-metrics-certs\") pod \"maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:03.868334 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.868287    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:03.868334 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.868333    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:03.868550 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.868351    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkvlg\" (UniqueName: \"kubernetes.io/projected/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-kube-api-access-bkvlg\") pod \"maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:03.868550 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.868394    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-metrics-certs\") pod \"maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:03.868550 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:19:03.868472    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-serving-cert: secret "maas-api-e2e-worker-w4-9360eb-serving-cert" not found
Oct 31 03:19:03.868706 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:19:03.868556    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-maas-api-tls podName:715cbcc7-1d2b-4fba-9527-4cf38503a4e3 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:04.368534308 +0000 UTC m=+685.929014030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-maas-api-tls") pod "maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr" (UID: "715cbcc7-1d2b-4fba-9527-4cf38503a4e3") : secret "maas-api-e2e-worker-w4-9360eb-serving-cert" not found
Oct 31 03:19:03.869136 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.869109    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:03.870841 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.870819    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-metrics-certs\") pod \"maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:03.923455 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:03.923375    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkvlg\" (UniqueName: \"kubernetes.io/projected/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-kube-api-access-bkvlg\") pod \"maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:04.372126 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:04.372078    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:04.374578 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:04.374547    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:04.573275 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:04.573237    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:04.604678 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:04.604643    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e370b8-a051-41f5-9641-9cd425369e48" path="/var/lib/kubelet/pods/21e370b8-a051-41f5-9641-9cd425369e48/volumes"
Oct 31 03:19:04.940095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:04.940006    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"]
Oct 31 03:19:04.943676 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:19:04.943645    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715cbcc7_1d2b_4fba_9527_4cf38503a4e3.slice/crio-804395e669c68422af8c550f0bebe85e0c1c50d39cc1d818204654d11f90735d WatchSource:0}: Error finding container 804395e669c68422af8c550f0bebe85e0c1c50d39cc1d818204654d11f90735d: Status 404 returned error can't find the container with id 804395e669c68422af8c550f0bebe85e0c1c50d39cc1d818204654d11f90735d
Oct 31 03:19:05.849438 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:05.849404    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr" event={"ID":"715cbcc7-1d2b-4fba-9527-4cf38503a4e3","Type":"ContainerStarted","Data":"804395e669c68422af8c550f0bebe85e0c1c50d39cc1d818204654d11f90735d"}
Oct 31 03:19:06.856013 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:06.855974    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr" event={"ID":"715cbcc7-1d2b-4fba-9527-4cf38503a4e3","Type":"ContainerStarted","Data":"0f3b2aa493c95487f9a77640fbf387e878a7e783f8c2a34ed63d52a9a6c95381"}
Oct 31 03:19:06.856440 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:06.856090    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:06.964380 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:06.964323    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr" podStartSLOduration=3.024341819 podStartE2EDuration="3.964308215s" totalImagesPullingTime="939.966396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:06.963631283 +0000 UTC m=+688.524111014" watchObservedRunningTime="2026-10-31 03:19:06.964308215 +0000 UTC m=+688.524787946"
Oct 31 03:19:12.866886 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:12.866847    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 03:19:14.908677 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:14.908641    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"]
Oct 31 03:19:14.912607 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:14.912581    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:14.961912 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:14.961851    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-d39257-serving-cert\""
Oct 31 03:19:15.011174 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.011133    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"]
Oct 31 03:19:15.072401 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.072363    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c85hm\" (UniqueName: \"kubernetes.io/projected/b54532f2-51ef-4ade-a041-b3c0c74105bf-kube-api-access-c85hm\") pod \"maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:15.072604 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.072564    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:15.072705 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.072685    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54532f2-51ef-4ade-a041-b3c0c74105bf-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:15.072762 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.072745    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-metrics-certs\") pod \"maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:15.174178 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.174076    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54532f2-51ef-4ade-a041-b3c0c74105bf-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:15.174178 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.174127    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-metrics-certs\") pod \"maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:15.174178 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.174155    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c85hm\" (UniqueName: \"kubernetes.io/projected/b54532f2-51ef-4ade-a041-b3c0c74105bf-kube-api-access-c85hm\") pod \"maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:15.174404 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.174285    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:15.174404 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:19:15.174386    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-serving-cert: secret "maas-api-e2e-worker-w1-d39257-serving-cert" not found
Oct 31 03:19:15.174469 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:19:15.174460    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-maas-api-tls podName:b54532f2-51ef-4ade-a041-b3c0c74105bf nodeName:}" failed. No retries permitted until 2026-10-31 03:19:15.674438804 +0000 UTC m=+697.234918513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-maas-api-tls") pod "maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw" (UID: "b54532f2-51ef-4ade-a041-b3c0c74105bf") : secret "maas-api-e2e-worker-w1-d39257-serving-cert" not found
Oct 31 03:19:15.174918 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.174870    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54532f2-51ef-4ade-a041-b3c0c74105bf-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:15.176708 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.176683    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-metrics-certs\") pod \"maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:15.230135 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.230097    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c85hm\" (UniqueName: \"kubernetes.io/projected/b54532f2-51ef-4ade-a041-b3c0c74105bf-kube-api-access-c85hm\") pod \"maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:15.679363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.679321    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:15.681724 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.681702    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:15.826299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:15.826259    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:16.173787 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:19:16.173745    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:19:16.195689 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:16.195658    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"]
Oct 31 03:19:16.198937 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:19:16.198906    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54532f2_51ef_4ade_a041_b3c0c74105bf.slice/crio-745e04bf7c46aa847f1bd981d080da7fc7dea5af2ed4598e879f2a9569cfb265 WatchSource:0}: Error finding container 745e04bf7c46aa847f1bd981d080da7fc7dea5af2ed4598e879f2a9569cfb265: Status 404 returned error can't find the container with id 745e04bf7c46aa847f1bd981d080da7fc7dea5af2ed4598e879f2a9569cfb265
Oct 31 03:19:16.904128 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:16.904089    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw" event={"ID":"b54532f2-51ef-4ade-a041-b3c0c74105bf","Type":"ContainerStarted","Data":"745e04bf7c46aa847f1bd981d080da7fc7dea5af2ed4598e879f2a9569cfb265"}
Oct 31 03:19:17.911150 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:17.911104    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw" event={"ID":"b54532f2-51ef-4ade-a041-b3c0c74105bf","Type":"ContainerStarted","Data":"ddd220bfdee4b0ca398de07f0277c44df54e72b44f6050c5c68ffad64b5b0d60"}
Oct 31 03:19:17.911651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:17.911184    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:18.024451 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:18.024391    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw" podStartSLOduration=3.123661443 podStartE2EDuration="4.024375698s" totalImagesPullingTime="900.714255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:18.023277634 +0000 UTC m=+699.583757391" watchObservedRunningTime="2026-10-31 03:19:18.024375698 +0000 UTC m=+699.584855429"
Oct 31 03:19:23.921660 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:23.921625    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 03:19:32.030213 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.030172    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"]
Oct 31 03:19:32.038172 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.038141    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.126768 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.126730    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.126768 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.126777    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.127058 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.126792    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj5gl\" (UniqueName: \"kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-kube-api-access-mj5gl\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.127058 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.126930    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.127058 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.126965    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.127058 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.126992    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.127272 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.127127    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.127272 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.127190    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.127272 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.127260    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.130327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.130307    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d39257-openshift-default-dockercfg-bm5z5\""
Oct 31 03:19:32.137190 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.137164    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"]
Oct 31 03:19:32.228692 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.228653    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.228692 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.228694    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.229014 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.228728    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.229014 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.228758    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.229014 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.228774    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj5gl\" (UniqueName: \"kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-kube-api-access-mj5gl\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.229014 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.228811    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.229234 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.229022    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.229234 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.229080    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.229234 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.229153    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.229234 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.229160    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.229234 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.229190    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.229553 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.229533    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.229553 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.229546    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.229865 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.229835    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.231439 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.231411    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.231838 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.231820    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.283427 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.283343    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj5gl\" (UniqueName: \"kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-kube-api-access-mj5gl\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.285307 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.285279    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.351816 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.351778    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:32.767348 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.767316    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"]
Oct 31 03:19:32.769663 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:19:32.769632    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021b11b1_eadc_4435_9bcc_9bac04de2bf8.slice/crio-9034737413dc87c84012caf0b19e49be0a3bac2ee5d6cd174314d198af42d1b2 WatchSource:0}: Error finding container 9034737413dc87c84012caf0b19e49be0a3bac2ee5d6cd174314d198af42d1b2: Status 404 returned error can't find the container with id 9034737413dc87c84012caf0b19e49be0a3bac2ee5d6cd174314d198af42d1b2
Oct 31 03:19:32.772208 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.772175    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:19:32.772292 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.772243    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:19:32.772292 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.772270    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:19:32.980558 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.980517    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm" event={"ID":"021b11b1-eadc-4435-9bcc-9bac04de2bf8","Type":"ContainerStarted","Data":"06ef4e51e9946f024fc7296e975d4a39ad28bdd48e94ddac6d66e210302dfce0"}
Oct 31 03:19:32.980729 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:32.980585    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm" event={"ID":"021b11b1-eadc-4435-9bcc-9bac04de2bf8","Type":"ContainerStarted","Data":"9034737413dc87c84012caf0b19e49be0a3bac2ee5d6cd174314d198af42d1b2"}
Oct 31 03:19:33.092538 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:33.092481    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm" podStartSLOduration=2.092466242 podStartE2EDuration="2.092466242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:33.091495866 +0000 UTC m=+714.651975617" watchObservedRunningTime="2026-10-31 03:19:33.092466242 +0000 UTC m=+714.652945973"
Oct 31 03:19:33.360969 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:33.355321    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:33.367044 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:33.367003    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 31 03:19:33.367317 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:33.367291    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm" podUID="021b11b1-eadc-4435-9bcc-9bac04de2bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 31 03:19:34.367922 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:34.367738    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 31 03:19:34.367922 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:34.367805    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm" podUID="021b11b1-eadc-4435-9bcc-9bac04de2bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 31 03:19:35.355379 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:35.355340    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 31 03:19:35.355641 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:35.355611    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm" podUID="021b11b1-eadc-4435-9bcc-9bac04de2bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 31 03:19:36.354905 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:36.353331    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 31 03:19:36.354905 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:36.353400    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm" podUID="021b11b1-eadc-4435-9bcc-9bac04de2bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 31 03:19:37.370760 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:37.370290    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 31 03:19:37.370760 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:37.370354    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm" podUID="021b11b1-eadc-4435-9bcc-9bac04de2bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 31 03:19:38.369096 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:38.369058    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 31 03:19:38.369385 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:38.369359    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm" podUID="021b11b1-eadc-4435-9bcc-9bac04de2bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 31 03:19:39.356476 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:39.356438    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:39.356959 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:39.356756    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:39.357540 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:39.357519    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 03:19:46.174716 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:19:46.174632    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:19:52.774965 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.774928    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"]
Oct 31 03:19:52.778727 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.778700    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.836007 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.835970    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd5b85\"/\"simulator-w3-dd5b85-kserve-self-signed-certs\""
Oct 31 03:19:52.856409 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.856367    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78mzh\" (UniqueName: \"kubernetes.io/projected/e8a80930-06e4-4b4e-9aec-79274f0961f0-kube-api-access-78mzh\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.856623 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.856476    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-model-cache\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.856623 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.856512    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a80930-06e4-4b4e-9aec-79274f0961f0-tls-certs\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.856623 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.856578    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-tmp-dir\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.856798 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.856661    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-home\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.856798 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.856722    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-dshm\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.875788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.875741    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd5b85\"/\"default-dockercfg-w824c\""
Oct 31 03:19:52.875788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.875776    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dd5b85\"/\"kube-root-ca.crt\""
Oct 31 03:19:52.876074 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.875954    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dd5b85\"/\"openshift-service-ca.crt\""
Oct 31 03:19:52.877290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.877269    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"]
Oct 31 03:19:52.957890 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.957829    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-model-cache\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.958089 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.957913    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a80930-06e4-4b4e-9aec-79274f0961f0-tls-certs\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.958089 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.957973    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-tmp-dir\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.958089 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.958019    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-home\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.958089 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.958057    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-dshm\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.958312 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.958111    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78mzh\" (UniqueName: \"kubernetes.io/projected/e8a80930-06e4-4b4e-9aec-79274f0961f0-kube-api-access-78mzh\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.958375 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.958352    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-tmp-dir\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.958431 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.958379    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-model-cache\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.958635 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.958610    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-home\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.960265 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.960243    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-dshm\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:52.960558 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:52.960542    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a80930-06e4-4b4e-9aec-79274f0961f0-tls-certs\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:53.014492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:53.014455    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78mzh\" (UniqueName: \"kubernetes.io/projected/e8a80930-06e4-4b4e-9aec-79274f0961f0-kube-api-access-78mzh\") pod \"simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") " pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:53.090511 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:53.090463    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:19:53.460617 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:53.460501    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"]
Oct 31 03:19:53.463398 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:19:53.463368    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8a80930_06e4_4b4e_9aec_79274f0961f0.slice/crio-e2698c08ef80db9bdef5ace5261253282821e469ce3c1f99dff06605f01214e8 WatchSource:0}: Error finding container e2698c08ef80db9bdef5ace5261253282821e469ce3c1f99dff06605f01214e8: Status 404 returned error can't find the container with id e2698c08ef80db9bdef5ace5261253282821e469ce3c1f99dff06605f01214e8
Oct 31 03:19:54.082390 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:54.082348    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd" event={"ID":"e8a80930-06e4-4b4e-9aec-79274f0961f0","Type":"ContainerStarted","Data":"e2698c08ef80db9bdef5ace5261253282821e469ce3c1f99dff06605f01214e8"}
Oct 31 03:19:56.094221 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:56.094185    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd" event={"ID":"e8a80930-06e4-4b4e-9aec-79274f0961f0","Type":"ContainerStarted","Data":"cd9970a7d7f35d370af07c7701c8d0f1ec967fc7d2128f118753da8c0b15be45"}
Oct 31 03:19:56.243353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:56.243287    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd" podStartSLOduration=2.565518939 podStartE2EDuration="4.243273175s" totalImagesPullingTime="1.677754236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:56.241950886 +0000 UTC m=+737.802430616" watchObservedRunningTime="2026-10-31 03:19:56.243273175 +0000 UTC m=+737.803752941"
Oct 31 03:19:57.091276 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:19:57.091232    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:20:03.091233 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:03.091190    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:20:03.105162 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:03.105126    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:20:03.141611 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:03.141579    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:20:13.566799 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.566754    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"]
Oct 31 03:20:13.571096 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.571071    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.640034 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.639990    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"kube-root-ca.crt\""
Oct 31 03:20:13.655313 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.655277    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tls-certs\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.655481 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.655322    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn798\" (UniqueName: \"kubernetes.io/projected/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-kube-api-access-mn798\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.655518 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.655470    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-dshm\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.655518 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.655512    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tmp-dir\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.655603 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.655584    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-home\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.655669 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.655630    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-model-cache\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.667947 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.667922    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"openshift-service-ca.crt\""
Oct 31 03:20:13.668178 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.668159    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"default-dockercfg-tmtvg\""
Oct 31 03:20:13.668275 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.668250    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"simulator-w1-d39257-kserve-self-signed-certs\""
Oct 31 03:20:13.669554 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.669534    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"]
Oct 31 03:20:13.757093 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.757047    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-model-cache\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.757290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.757124    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tls-certs\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.757290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.757157    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn798\" (UniqueName: \"kubernetes.io/projected/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-kube-api-access-mn798\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.757290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.757244    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-dshm\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.757290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.757268    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tmp-dir\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.757581 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.757314    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-home\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.757581 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.757556    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-model-cache\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.757703 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.757678    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-home\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.757943 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.757919    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tmp-dir\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.759688 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.759661    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-dshm\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.759921 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.759902    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tls-certs\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.811864 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.811832    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn798\" (UniqueName: \"kubernetes.io/projected/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-kube-api-access-mn798\") pod \"simulator-w1-d39257-kserve-559566dd5d-74jrt\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") " pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:13.882920 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:13.882868    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:14.256031 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:14.255947    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"]
Oct 31 03:20:14.258720 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:20:14.258680    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b1f3a9b_f45a_44ab_9ee7_8c91aa066f5a.slice/crio-eb4fff0cfb75d411362b4e2a2edbbe3942b4c6778f53702d5cfaca9de79dba9d WatchSource:0}: Error finding container eb4fff0cfb75d411362b4e2a2edbbe3942b4c6778f53702d5cfaca9de79dba9d: Status 404 returned error can't find the container with id eb4fff0cfb75d411362b4e2a2edbbe3942b4c6778f53702d5cfaca9de79dba9d
Oct 31 03:20:15.191543 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:15.191485    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt" event={"ID":"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a","Type":"ContainerStarted","Data":"6a8f95ba370bde5187643f288281db6ca064677126d024b8de9a453d9c2f7827"}
Oct 31 03:20:15.191990 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:15.191555    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt" event={"ID":"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a","Type":"ContainerStarted","Data":"eb4fff0cfb75d411362b4e2a2edbbe3942b4c6778f53702d5cfaca9de79dba9d"}
Oct 31 03:20:15.301254 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:15.301199    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt" podStartSLOduration=2.301186112 podStartE2EDuration="2.301186112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:15.300981128 +0000 UTC m=+756.861460881" watchObservedRunningTime="2026-10-31 03:20:15.301186112 +0000 UTC m=+756.861665842"
Oct 31 03:20:15.883149 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:15.883104    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:16.175954 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:20:16.175853    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:20:23.883963 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:23.883921    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:23.896663 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:23.896630    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:24.244595 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:24.244519    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:20:45.634112 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.634075    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"]
Oct 31 03:20:45.640139 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.640112    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.736554 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.736521    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd5b85\"/\"premium-w3-dd5b85-kserve-self-signed-certs\""
Oct 31 03:20:45.737960 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.737936    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"]
Oct 31 03:20:45.761041 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.761003    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-home\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.761219 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.761050    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7286f18c-2456-47f7-a04d-078677b37a27-tls-certs\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.761219 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.761132    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-dshm\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.761219 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.761163    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-model-cache\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.761219 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.761185    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-tmp-dir\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.761384 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.761369    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w5lr\" (UniqueName: \"kubernetes.io/projected/7286f18c-2456-47f7-a04d-078677b37a27-kube-api-access-8w5lr\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.862294 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.862252    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-dshm\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.862294 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.862293    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-model-cache\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.862517 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.862430    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-tmp-dir\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.862517 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.862505    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w5lr\" (UniqueName: \"kubernetes.io/projected/7286f18c-2456-47f7-a04d-078677b37a27-kube-api-access-8w5lr\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.862745 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.862631    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-home\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.862745 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.862691    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7286f18c-2456-47f7-a04d-078677b37a27-tls-certs\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.862982 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.862785    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-model-cache\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.862982 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.862866    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-tmp-dir\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.863105 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.863005    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-home\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.864635 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.864610    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-dshm\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.865033 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.865014    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7286f18c-2456-47f7-a04d-078677b37a27-tls-certs\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.920795 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.920720    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w5lr\" (UniqueName: \"kubernetes.io/projected/7286f18c-2456-47f7-a04d-078677b37a27-kube-api-access-8w5lr\") pod \"premium-w3-dd5b85-kserve-544fbc7799-bjqsg\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") " pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:45.953542 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:45.953497    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:46.175664 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:20:46.175564    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:20:46.327534 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:46.327507    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"]
Oct 31 03:20:46.329953 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:20:46.329915    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7286f18c_2456_47f7_a04d_078677b37a27.slice/crio-fab54138ae2ad1a919e5d019c6a1431c256e0083351438568ce0ee96436ff7f8 WatchSource:0}: Error finding container fab54138ae2ad1a919e5d019c6a1431c256e0083351438568ce0ee96436ff7f8: Status 404 returned error can't find the container with id fab54138ae2ad1a919e5d019c6a1431c256e0083351438568ce0ee96436ff7f8
Oct 31 03:20:46.348270 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:46.348238    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" event={"ID":"7286f18c-2456-47f7-a04d-078677b37a27","Type":"ContainerStarted","Data":"fab54138ae2ad1a919e5d019c6a1431c256e0083351438568ce0ee96436ff7f8"}
Oct 31 03:20:47.354643 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:47.354602    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" event={"ID":"7286f18c-2456-47f7-a04d-078677b37a27","Type":"ContainerStarted","Data":"fbdba167ed831140cb7d8c45547e32e26b74ed5ffeb32496f2b681d3ddda5541"}
Oct 31 03:20:47.465533 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:47.465454    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" podStartSLOduration=2.465437294 podStartE2EDuration="2.465437294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:47.464455355 +0000 UTC m=+789.024935092" watchObservedRunningTime="2026-10-31 03:20:47.465437294 +0000 UTC m=+789.025917026"
Oct 31 03:20:47.954612 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:47.954573    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:55.954604 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:55.954565    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:55.973366 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:55.973331    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:20:56.410566 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:20:56.410529    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:21:16.175481 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:21:16.175386    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:21:25.276131 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.276088    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"]
Oct 31 03:21:25.279705 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.279680    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.359420 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.359372    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-model-cache\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.359607 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.359434    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-home\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.359607 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.359501    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-dshm\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.359607 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.359564    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72w8\" (UniqueName: \"kubernetes.io/projected/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-kube-api-access-v72w8\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.359738 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.359649    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tls-certs\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.359738 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.359719    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tmp-dir\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.376347 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.376313    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"premium-w1-d39257-kserve-self-signed-certs\""
Oct 31 03:21:25.378464 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.378441    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"]
Oct 31 03:21:25.460706 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.460670    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-model-cache\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.460706 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.460709    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-home\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.461008 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.460734    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-dshm\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.461008 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.460782    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72w8\" (UniqueName: \"kubernetes.io/projected/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-kube-api-access-v72w8\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.461008 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.460823    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tls-certs\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.461008 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.460911    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tmp-dir\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.461241 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.461186    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-model-cache\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.461300 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.461251    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-home\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.461300 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.461276    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tmp-dir\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.463360 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.463338    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-dshm\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.463618 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.463596    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tls-certs\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.515412 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.515371    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72w8\" (UniqueName: \"kubernetes.io/projected/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-kube-api-access-v72w8\") pod \"premium-w1-d39257-kserve-754b5bb4c-f2sps\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") " pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.591934 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.591871    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:25.976833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:25.976749    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"]
Oct 31 03:21:25.980644 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:21:25.980613    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d1ad969_4b4b_4178_a5a9_7f76d8cd2150.slice/crio-aa270eec466e8d5c3c1e1874d67642538f8fd16cfa45630656f79c0790f38ec5 WatchSource:0}: Error finding container aa270eec466e8d5c3c1e1874d67642538f8fd16cfa45630656f79c0790f38ec5: Status 404 returned error can't find the container with id aa270eec466e8d5c3c1e1874d67642538f8fd16cfa45630656f79c0790f38ec5
Oct 31 03:21:26.558962 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:26.558921    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" event={"ID":"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150","Type":"ContainerStarted","Data":"de39d696103a67b4320410a5ab7e13cb91f240cf1bc781ccdb98c0087cbbd1f9"}
Oct 31 03:21:26.559363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:26.558981    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" event={"ID":"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150","Type":"ContainerStarted","Data":"aa270eec466e8d5c3c1e1874d67642538f8fd16cfa45630656f79c0790f38ec5"}
Oct 31 03:21:26.592859 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:26.592826    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:26.669558 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:26.669502    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" podStartSLOduration=1.669481598 podStartE2EDuration="1.669481598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:26.66781172 +0000 UTC m=+828.228291471" watchObservedRunningTime="2026-10-31 03:21:26.669481598 +0000 UTC m=+828.229961329"
Oct 31 03:21:35.592783 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:35.592738    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:35.606117 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:35.606087    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:35.617447 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:21:35.617420    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:21:46.174712 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:21:46.174671    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:22:16.176156 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:22:16.176116    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:22:46.175868 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:22:46.175784    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:23:00.448855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.448762    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"]
Oct 31 03:23:00.452549 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.452525    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.520781 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.520745    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tls-certs\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.520964 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.520806    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-model-cache\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.521009 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.520970    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-home\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.521044 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.521009    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-dshm\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.521287 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.521261    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tmp-dir\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.521421 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.521326    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx2gj\" (UniqueName: \"kubernetes.io/projected/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-kube-api-access-fx2gj\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.549590 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.549556    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"e2e-conflict-a-3b9043d1-kserve-self-signed-certs\""
Oct 31 03:23:00.551109 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.551079    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"]
Oct 31 03:23:00.622736 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.622703    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tmp-dir\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.622964 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.622760    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx2gj\" (UniqueName: \"kubernetes.io/projected/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-kube-api-access-fx2gj\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.622964 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.622790    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tls-certs\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.622964 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.622821    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-model-cache\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.622964 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.622863    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-home\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.622964 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.622913    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-dshm\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.623286 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.623141    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tmp-dir\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.623286 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.623211    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-model-cache\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.623386 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.623290    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-home\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.625170 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.625150    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-dshm\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.625459 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.625438    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tls-certs\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.678067 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.678013    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx2gj\" (UniqueName: \"kubernetes.io/projected/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-kube-api-access-fx2gj\") pod \"e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:00.764496 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:00.764400    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:01.149125 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:01.149092    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"]
Oct 31 03:23:01.152399 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:23:01.152364    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20a3fef2_03e5_4fd0_86e1_1ae47cdfc642.slice/crio-502557bd1e23f6fcfa5d07830def5418c6281bb52627b6d4a113edae81a41afd WatchSource:0}: Error finding container 502557bd1e23f6fcfa5d07830def5418c6281bb52627b6d4a113edae81a41afd: Status 404 returned error can't find the container with id 502557bd1e23f6fcfa5d07830def5418c6281bb52627b6d4a113edae81a41afd
Oct 31 03:23:02.022444 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:02.022403    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg" event={"ID":"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642","Type":"ContainerStarted","Data":"b70fda5a64b5c5f2540e7cb1a248dbf8b73e17feef2275967049a9408068c4d4"}
Oct 31 03:23:02.023092 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:02.023051    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg" event={"ID":"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642","Type":"ContainerStarted","Data":"502557bd1e23f6fcfa5d07830def5418c6281bb52627b6d4a113edae81a41afd"}
Oct 31 03:23:02.132069 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:02.132012    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg" podStartSLOduration=2.131996118 podStartE2EDuration="2.131996118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:02.130767547 +0000 UTC m=+923.691247279" watchObservedRunningTime="2026-10-31 03:23:02.131996118 +0000 UTC m=+923.692475848"
Oct 31 03:23:02.765001 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:02.764962    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:08.195179 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.195144    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"]
Oct 31 03:23:08.198990 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.198965    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.306978 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.306940    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.307173 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.307074    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2161a56-053a-4467-8a42-608e85bcedb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.307247 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.307171    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.307247 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.307213    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.307247 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.307242    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2161a56-053a-4467-8a42-608e85bcedb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.307436 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.307408    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2r6\" (UniqueName: \"kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-kube-api-access-9t2r6\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.307489 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.307459    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.307600 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.307586    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.307655 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.307617    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.402064 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.402031    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"]
Oct 31 03:23:08.408645 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.408614    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2r6\" (UniqueName: \"kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-kube-api-access-9t2r6\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.408645 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.408651    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.408927 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.408719    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.408927 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.408745    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.408927 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.408780    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.408927 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.408827    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2161a56-053a-4467-8a42-608e85bcedb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.408927 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.408867    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.408927 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.408910    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.409226 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.408936    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2161a56-053a-4467-8a42-608e85bcedb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.409545 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.409511    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.409716 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.409685    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.409847 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.409825    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.409943 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.409933    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2161a56-053a-4467-8a42-608e85bcedb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.410031 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.410011    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.412010 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.411984    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2161a56-053a-4467-8a42-608e85bcedb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.415387 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.415357    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.463454 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.463373    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.463615 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.463487    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2r6\" (UniqueName: \"kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-kube-api-access-9t2r6\") pod \"maas-default-gateway-openshift-default-6886b96f9-flbbt\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.512613 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.512577    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:08.932466 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.932435    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"]
Oct 31 03:23:08.937538 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.937504    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:23:08.937639 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.937574    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:23:08.937639 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:08.937604    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:23:09.058578 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:09.058538    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" event={"ID":"a2161a56-053a-4467-8a42-608e85bcedb9","Type":"ContainerStarted","Data":"91595f7a2f621bd9533a6bf34679350961ed43bb20bc661b55af6a1876367703"}
Oct 31 03:23:09.058756 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:09.058606    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" event={"ID":"a2161a56-053a-4467-8a42-608e85bcedb9","Type":"ContainerStarted","Data":"586718237c1932a652f118555b8aa0af2ef2d9dc3754a88e8542b69381076068"}
Oct 31 03:23:09.169627 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:09.169565    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" podStartSLOduration=1.16954972 podStartE2EDuration="1.16954972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:09.168426873 +0000 UTC m=+930.728906625" watchObservedRunningTime="2026-10-31 03:23:09.16954972 +0000 UTC m=+930.730029504"
Oct 31 03:23:09.513555 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:09.513503    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:10.514395 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:10.514356    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-flbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:23:10.514806 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:10.514416    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" podUID="a2161a56-053a-4467-8a42-608e85bcedb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 31 03:23:10.771963 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:10.767389    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:10.783645 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:10.783607    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:11.092049 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:11.092020    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:23:11.513645 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:11.513549    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-flbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:23:11.513645 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:11.513608    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" podUID="a2161a56-053a-4467-8a42-608e85bcedb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 31 03:23:12.513205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:12.513170    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-flbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:23:12.513623 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:12.513228    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" podUID="a2161a56-053a-4467-8a42-608e85bcedb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 31 03:23:13.513461 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:13.513426    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-flbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:23:13.513890 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:13.513498    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" podUID="a2161a56-053a-4467-8a42-608e85bcedb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 31 03:23:14.513913 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.513846    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-flbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:23:14.514567 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.513948    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" podUID="a2161a56-053a-4467-8a42-608e85bcedb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 31 03:23:14.728513 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.728477    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"]
Oct 31 03:23:14.732531 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.732506    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.788341 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.788245    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9360eb-openshift-default-dockercfg-lsbz7\""
Oct 31 03:23:14.835365 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.835309    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"]
Oct 31 03:23:14.897552 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.897515    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.897720 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.897571    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.897778 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.897749    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.897819 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.897800    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.897917 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.897902    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.897982 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.897965    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.898019 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.898005    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.898063 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.898033    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9647\" (UniqueName: \"kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-kube-api-access-d9647\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.898128 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.898109    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.999481 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.999441    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.999481 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.999497    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.999754 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.999544    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.999754 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.999604    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.999754 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.999632    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.999754 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.999665    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9647\" (UniqueName: \"kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-kube-api-access-d9647\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.999754 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.999695    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:14.999754 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.999731    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:15.000111 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.999778    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:15.000111 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:14.999999    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:15.000111 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.000026    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:15.000377 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.000357    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:15.000458 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.000406    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:15.000508 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.000489    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:15.002020 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.001999    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:15.002113 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.002097    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:15.053767 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.053689    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:15.053988 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.053962    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9647\" (UniqueName: \"kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-kube-api-access-d9647\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:15.346702 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.346663    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:15.513471 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.513436    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-flbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:23:15.513669 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.513513    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" podUID="a2161a56-053a-4467-8a42-608e85bcedb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 31 03:23:15.768932 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.768903    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"]
Oct 31 03:23:15.779301 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.779237    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:23:15.779453 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.779343    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:23:15.779453 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:15.779389    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:23:16.113574 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:16.113528    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" event={"ID":"51ec02cc-eaaa-4913-8904-6a97d51a0d46","Type":"ContainerStarted","Data":"6d89376a0bad26cb07c7eb8aed302741ffb3247cb35060e9795f37d5dc0bc5ab"}
Oct 31 03:23:16.113801 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:16.113601    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" event={"ID":"51ec02cc-eaaa-4913-8904-6a97d51a0d46","Type":"ContainerStarted","Data":"51ebd421c132c923065a422ed903d9aa3685838a886313dc64785d07fd417c1d"}
Oct 31 03:23:16.190996 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:23:16.190950    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:23:16.225376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:16.225146    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" podStartSLOduration=2.225125975 podStartE2EDuration="2.225125975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:16.224153493 +0000 UTC m=+937.784633263" watchObservedRunningTime="2026-10-31 03:23:16.225125975 +0000 UTC m=+937.785605707"
Oct 31 03:23:16.347422 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:16.347278    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:16.349144 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:16.349107    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:23:16.349356 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:16.349178    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" podUID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 03:23:16.514476 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:16.514362    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-flbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:23:16.514476 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:16.514438    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" podUID="a2161a56-053a-4467-8a42-608e85bcedb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 31 03:23:17.352906 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:17.351348    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:23:17.352906 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:17.351410    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" podUID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 03:23:18.279289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:18.279253    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:18.279569 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:18.279547    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:18.280540 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:18.280521    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:23:18.355904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:18.354369    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:23:18.355904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:18.354432    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" podUID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 03:23:19.352901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:19.351352    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:23:19.352901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:19.351419    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" podUID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 03:23:20.348562 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:20.348062    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:23:20.348562 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:20.348131    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" podUID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 03:23:21.358903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:21.351337    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:23:21.358903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:21.351409    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" podUID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 03:23:22.347510 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:22.347463    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:23:22.347733 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:22.347521    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" podUID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 03:23:23.291562 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.291525    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"]
Oct 31 03:23:23.295610 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.295581    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.325539 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.325501    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.325740 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.325562    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncl7d\" (UniqueName: \"kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-kube-api-access-ncl7d\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.325740 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.325655    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.325740 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.325720    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/057c4b5b-d48f-413f-9a9c-43db3e06f380-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.325952 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.325822    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.325952 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.325867    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.325952 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.325918    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.325952 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.325946    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.326131 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.326050    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.352177 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.352140    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:23.352614 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.352576    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:23.353481 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.353456    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:23:23.394664 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.394627    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"]
Oct 31 03:23:23.427386 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.427331    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/057c4b5b-d48f-413f-9a9c-43db3e06f380-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.427599 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.427411    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.427599 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.427523    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.427599 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.427575    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.427758 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.427713    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.427871 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.427803    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.427871 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.427834    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.428020 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.427922    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncl7d\" (UniqueName: \"kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-kube-api-access-ncl7d\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.428020 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.427934    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.428020 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.427987    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.428020 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.428008    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.428377 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.428350    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/057c4b5b-d48f-413f-9a9c-43db3e06f380-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.428462 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.428359    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.428501 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.428463    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.430232 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.430201    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.430512 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.430492    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.481712 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.481670    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.481920 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.481779    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncl7d\" (UniqueName: \"kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-kube-api-access-ncl7d\") pod \"maas-default-gateway-openshift-default-6886b96f9-76m8v\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:23.609911 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:23.609865    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:24.026114 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:24.026034    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"]
Oct 31 03:23:24.064268 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:23:24.064233    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057c4b5b_d48f_413f_9a9c_43db3e06f380.slice/crio-821cbbd9299603b2dee74f59bb7a23960e1f2e4b6002b57e77d03cd7756a90b8 WatchSource:0}: Error finding container 821cbbd9299603b2dee74f59bb7a23960e1f2e4b6002b57e77d03cd7756a90b8: Status 404 returned error can't find the container with id 821cbbd9299603b2dee74f59bb7a23960e1f2e4b6002b57e77d03cd7756a90b8
Oct 31 03:23:24.066819 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:24.066781    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:23:24.066934 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:24.066865    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:23:24.066934 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:24.066911    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:23:24.178278 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:24.178235    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" event={"ID":"057c4b5b-d48f-413f-9a9c-43db3e06f380","Type":"ContainerStarted","Data":"270f6af21b472f620d09a785077c5df3ae4e4dcf1dcc6335dbcd3b80dad15637"}
Oct 31 03:23:24.178479 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:24.178304    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" event={"ID":"057c4b5b-d48f-413f-9a9c-43db3e06f380","Type":"ContainerStarted","Data":"821cbbd9299603b2dee74f59bb7a23960e1f2e4b6002b57e77d03cd7756a90b8"}
Oct 31 03:23:24.289615 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:24.289510    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" podStartSLOduration=1.2894830640000001 podStartE2EDuration="1.289483064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:24.288996301 +0000 UTC m=+945.849476058" watchObservedRunningTime="2026-10-31 03:23:24.289483064 +0000 UTC m=+945.849962795"
Oct 31 03:23:24.610192 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:24.610165    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:25.612067 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:25.612030    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-76m8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 03:23:25.612575 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:25.612084    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" podUID="057c4b5b-d48f-413f-9a9c-43db3e06f380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 03:23:26.610869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:26.610832    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-76m8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 03:23:26.611106 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:26.610913    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" podUID="057c4b5b-d48f-413f-9a9c-43db3e06f380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 03:23:27.611163 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:27.611127    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-76m8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 03:23:27.611663 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:27.611180    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" podUID="057c4b5b-d48f-413f-9a9c-43db3e06f380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 03:23:28.610505 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:28.610428    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-76m8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 03:23:28.610723 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:28.610533    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" podUID="057c4b5b-d48f-413f-9a9c-43db3e06f380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 03:23:29.610295 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:29.610254    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-76m8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 03:23:29.610817 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:29.610316    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" podUID="057c4b5b-d48f-413f-9a9c-43db3e06f380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 03:23:30.611108 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:30.611066    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-76m8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 03:23:30.611611 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:30.611124    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" podUID="057c4b5b-d48f-413f-9a9c-43db3e06f380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 03:23:31.610921 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:31.610867    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-76m8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 03:23:31.611114 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:31.610935    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" podUID="057c4b5b-d48f-413f-9a9c-43db3e06f380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 03:23:32.615986 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:32.615947    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:32.617528 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:32.617502    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:32.618112 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:32.618087    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:23:33.868347 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:33.868303    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"]
Oct 31 03:23:33.872445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:33.872414    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:33.967567 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:33.967525    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:33.967766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:33.967598    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jbvp\" (UniqueName: \"kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-kube-api-access-4jbvp\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:33.967766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:33.967629    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:33.967766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:33.967655    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:33.967766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:33.967682    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:33.967766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:33.967741    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:33.967766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:33.967763    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:33.968169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:33.967790    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:33.968169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:33.967819    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.069036 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.068997    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.069036 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.069041    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.069332 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.069158    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.069332 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.069234    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jbvp\" (UniqueName: \"kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-kube-api-access-4jbvp\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.069645 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.069267    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.069728 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.069683    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.069801 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.069569    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.069861 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.069791    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.069994 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.069973    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.070054 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.069532    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.070117 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.070073    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.070176 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.070166    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.070235 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.070191    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.071000 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.070972    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.072731 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.072701    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.073093 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.073057    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.083056 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.082964    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"]
Oct 31 03:23:34.128010 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.127535    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.138360 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.138008    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jbvp\" (UniqueName: \"kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-kube-api-access-4jbvp\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.214929 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.214563    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:34.762910 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.758791    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"]
Oct 31 03:23:34.773910 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:23:34.772141    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b97d07f_dda8_485e_9377_61b8df3ffbb5.slice/crio-5aba828de417b7645d1872e4e18ccfba70a8c144b20d0ebaaa8853d49d4db153 WatchSource:0}: Error finding container 5aba828de417b7645d1872e4e18ccfba70a8c144b20d0ebaaa8853d49d4db153: Status 404 returned error can't find the container with id 5aba828de417b7645d1872e4e18ccfba70a8c144b20d0ebaaa8853d49d4db153
Oct 31 03:23:34.777805 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.777768    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:23:34.778169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.778143    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:23:34.784360 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:34.784323    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:23:35.233901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:35.233169    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hbjsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:23:35.233901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:35.233270    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" podUID="c2b20bf0-72db-4618-b347-c8db2b887b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:23:35.269902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:35.265777    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww" event={"ID":"5b97d07f-dda8-485e-9377-61b8df3ffbb5","Type":"ContainerStarted","Data":"e1a113cc32101ba2d0018191c97f2ba662f0dd70f29dbbacb02c19015986ce9b"}
Oct 31 03:23:35.269902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:35.265857    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww" event={"ID":"5b97d07f-dda8-485e-9377-61b8df3ffbb5","Type":"ContainerStarted","Data":"5aba828de417b7645d1872e4e18ccfba70a8c144b20d0ebaaa8853d49d4db153"}
Oct 31 03:23:36.216085 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:36.216048    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:36.217951 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:36.217922    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 03:23:36.218068 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:36.217975    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww" podUID="5b97d07f-dda8-485e-9377-61b8df3ffbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 03:23:37.215208 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:37.215174    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 03:23:37.215613 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:37.215227    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww" podUID="5b97d07f-dda8-485e-9377-61b8df3ffbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 03:23:38.215768 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:38.215729    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 03:23:38.216296 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:38.215794    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww" podUID="5b97d07f-dda8-485e-9377-61b8df3ffbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 03:23:39.215594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:39.215551    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 03:23:39.215795 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:39.215623    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww" podUID="5b97d07f-dda8-485e-9377-61b8df3ffbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 03:23:40.216222 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:40.216178    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 03:23:40.216651 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:40.216255    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww" podUID="5b97d07f-dda8-485e-9377-61b8df3ffbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 03:23:41.216287 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:41.216248    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 03:23:41.216690 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:41.216310    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww" podUID="5b97d07f-dda8-485e-9377-61b8df3ffbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 03:23:42.377568 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.377526    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:42.378340 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.378317    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:42.379370 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.379342    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:23:42.538417 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.538359    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww" podStartSLOduration=9.538344091999999 podStartE2EDuration="9.538344092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:35.387172262 +0000 UTC m=+956.947651993" watchObservedRunningTime="2026-10-31 03:23:42.538344092 +0000 UTC m=+964.098823822"
Oct 31 03:23:42.605729 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.605698    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"]
Oct 31 03:23:42.610812 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.610791    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.662536 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.662406    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9360eb\"/\"kube-root-ca.crt\""
Oct 31 03:23:42.662695 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.662571    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9360eb\"/\"openshift-service-ca.crt\""
Oct 31 03:23:42.663059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.663039    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9360eb\"/\"default-dockercfg-w6g9t\""
Oct 31 03:23:42.665053 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.665029    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"]
Oct 31 03:23:42.681599 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.681563    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9360eb\"/\"distinct-w4-9360eb-kserve-self-signed-certs\""
Oct 31 03:23:42.741156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.741115    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tls-certs\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.741430 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.741255    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfnkc\" (UniqueName: \"kubernetes.io/projected/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-kube-api-access-gfnkc\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.741430 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.741345    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-dshm\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.741430 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.741378    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-model-cache\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.741623 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.741456    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tmp-dir\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.741623 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.741529    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-home\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.842410 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.842375    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tmp-dir\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.842623 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.842435    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-home\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.842623 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.842491    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tls-certs\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.842623 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.842572    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfnkc\" (UniqueName: \"kubernetes.io/projected/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-kube-api-access-gfnkc\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.842623 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.842616    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-dshm\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.842854 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.842638    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-model-cache\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.842932 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.842852    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tmp-dir\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.843358 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.843329    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-home\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.843464 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.843373    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-model-cache\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.845409 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.845383    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-dshm\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.846144 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.846123    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tls-certs\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.897268 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.897228    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfnkc\" (UniqueName: \"kubernetes.io/projected/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-kube-api-access-gfnkc\") pod \"distinct-w4-9360eb-kserve-5799546b88-t766p\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:42.923800 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:42.923700    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:43.327141 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:43.327119    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"]
Oct 31 03:23:44.328888 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:44.328832    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" event={"ID":"72b65a20-6ba2-4f42-99e8-60d7a070aa5f","Type":"ContainerStarted","Data":"935c796299a40f1c605fff942b6224863a97b4673d614da3a537856b1ac6719f"}
Oct 31 03:23:44.329351 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:44.328897    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" event={"ID":"72b65a20-6ba2-4f42-99e8-60d7a070aa5f","Type":"ContainerStarted","Data":"9c3b907b4b97a630de28905e3b851f531d3548e2b958987d4bd40a10167ddc53"}
Oct 31 03:23:44.438707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:44.438647    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" podStartSLOduration=2.438630096 podStartE2EDuration="2.438630096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:44.437102302 +0000 UTC m=+965.997582037" watchObservedRunningTime="2026-10-31 03:23:44.438630096 +0000 UTC m=+965.999109827"
Oct 31 03:23:44.924625 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:44.924586    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:46.175467 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:23:46.175425    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:23:52.924144 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:52.924096    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:52.937579 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:52.937552    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:23:53.388803 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:23:53.388765    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:24:07.517448 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.517400    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"]
Oct 31 03:24:07.521600 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.521573    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.571387 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.571351    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7778d0-openshift-default-dockercfg-c48rb\""
Oct 31 03:24:07.624415 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.624376    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"]
Oct 31 03:24:07.629947 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.629871    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.630127 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.630048    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.630127 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.630096    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.630248 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.630204    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.630308 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.630246    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.630357 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.630303    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.630357 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.630329    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.630510 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.630485    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88nbt\" (UniqueName: \"kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-kube-api-access-88nbt\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.630569 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.630553    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.732054 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.732015    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88nbt\" (UniqueName: \"kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-kube-api-access-88nbt\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.732276 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.732079    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.732276 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.732178    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.732276 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.732232    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.732449 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.732377    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.732449 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.732426    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.732556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.732460    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.732556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.732488    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.732556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.732514    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.732556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.732547    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.732768 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.732599    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.732860 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.732834    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.732960 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.732901    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.733278 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.733262    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.734700 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.734678    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.734779 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.734722    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.786733 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.786667    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.786873 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.786732    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88nbt\" (UniqueName: \"kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-kube-api-access-88nbt\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:07.836402 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:07.836364    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:08.210365 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:08.210335    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"]
Oct 31 03:24:08.212363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:24:08.212322    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03920f9_a5dc_4eb0_8e03_4f5fb85f80a1.slice/crio-d0a88937c3dc85192f8bbae8b747e371a8cea8289194930e83150b59172cfbd9 WatchSource:0}: Error finding container d0a88937c3dc85192f8bbae8b747e371a8cea8289194930e83150b59172cfbd9: Status 404 returned error can't find the container with id d0a88937c3dc85192f8bbae8b747e371a8cea8289194930e83150b59172cfbd9
Oct 31 03:24:08.214537 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:08.214506    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:24:08.214627 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:08.214571    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:24:08.214627 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:08.214602    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:24:08.456430 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:08.456385    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs" event={"ID":"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1","Type":"ContainerStarted","Data":"9e34f15668c40bdcea4703d6abe1d6984ff1795cfa7f06df0e4a568acfa89cb5"}
Oct 31 03:24:08.456600 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:08.456511    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs" event={"ID":"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1","Type":"ContainerStarted","Data":"d0a88937c3dc85192f8bbae8b747e371a8cea8289194930e83150b59172cfbd9"}
Oct 31 03:24:08.570035 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:08.569919    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs" podStartSLOduration=1.569902846 podStartE2EDuration="1.569902846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:08.567673956 +0000 UTC m=+990.128153707" watchObservedRunningTime="2026-10-31 03:24:08.569902846 +0000 UTC m=+990.130382574"
Oct 31 03:24:08.837149 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:08.837110    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:09.838719 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:09.838683    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:24:09.839168 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:09.838736    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs" podUID="d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 03:24:10.837509 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:10.837463    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:24:10.837728 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:10.837534    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs" podUID="d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 03:24:11.837330 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:11.837294    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:24:11.837766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:11.837354    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs" podUID="d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 03:24:12.837361 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:12.837323    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:24:12.837860 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:12.837400    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs" podUID="d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 03:24:13.837291 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:13.837250    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:24:13.837487 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:13.837337    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs" podUID="d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 03:24:14.837645 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:14.837603    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:24:14.838216 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:14.837680    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs" podUID="d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 03:24:15.841456 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:15.841422    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:15.841967 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:15.841735    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:15.842524 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:15.842508    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 03:24:16.179903 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:24:16.179792    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:24:46.176089 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:24:46.176046    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:24:49.439907 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.439857    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"]
Oct 31 03:24:49.444819 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.444799    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.613084 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.613045    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.613258 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.613100    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgc54\" (UniqueName: \"kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-kube-api-access-qgc54\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.613258 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.613231    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.613335 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.613293    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.613374 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.613356    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.613409 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.613385    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.613441 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.613409    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.613474 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.613451    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.613580 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.613566    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.649807 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.649769    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"]
Oct 31 03:24:49.714944 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.714848    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.714944 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.714906    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.714944 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.714929    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.715214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.714952    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.715214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.714978    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.715214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.715068    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.715214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.715120    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.715214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.715166    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgc54\" (UniqueName: \"kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-kube-api-access-qgc54\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.715475 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.715227    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.715475 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.715364    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.715475 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.715397    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.715616 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.715522    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.715616 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.715600    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.715984 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.715965    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.717397 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.717373    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.717548 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.717530    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.769575 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.769539    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:49.769575 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:49.769558    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgc54\" (UniqueName: \"kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-kube-api-access-qgc54\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:50.057930 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:50.057832    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:50.476938 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:50.476740    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"]
Oct 31 03:24:50.479702 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:24:50.479669    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba281b6_72bf_46f8_b5ed_c1c10f1bc980.slice/crio-b257e4a76da34e9a038d55c96aa6c35dd8bac53da589345ac89424e64ae56f91 WatchSource:0}: Error finding container b257e4a76da34e9a038d55c96aa6c35dd8bac53da589345ac89424e64ae56f91: Status 404 returned error can't find the container with id b257e4a76da34e9a038d55c96aa6c35dd8bac53da589345ac89424e64ae56f91
Oct 31 03:24:50.482101 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:50.482070    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:24:50.482207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:50.482153    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:24:50.482207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:50.482197    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:24:50.768396 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:50.768304    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" event={"ID":"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980","Type":"ContainerStarted","Data":"9c845cf0b04409c5dae2d87edf67fd050c9a9e8281194b50bca899ebc52b5567"}
Oct 31 03:24:50.768396 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:50.768371    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" event={"ID":"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980","Type":"ContainerStarted","Data":"b257e4a76da34e9a038d55c96aa6c35dd8bac53da589345ac89424e64ae56f91"}
Oct 31 03:24:50.881937 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:50.881864    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" podStartSLOduration=1.8818492839999998 podStartE2EDuration="1.881849284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:50.880998081 +0000 UTC m=+1032.441477837" watchObservedRunningTime="2026-10-31 03:24:50.881849284 +0000 UTC m=+1032.442329015"
Oct 31 03:24:51.058268 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:51.058185    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:51.059643 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:51.059618    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:24:51.059766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:51.059666    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 03:24:52.062835 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:52.062355    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:24:52.062835 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:52.062422    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 03:24:53.065901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:53.064999    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:24:53.065901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:53.065068    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 03:24:54.069350 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:54.069310    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:24:54.070019 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:54.069987    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 03:24:55.065902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:55.060276    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:24:55.065902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:55.060345    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 03:24:56.060846 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:56.060805    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:24:56.061469 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:56.061433    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 03:24:57.059302 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:57.059258    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:24:57.059498 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:57.059325    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 03:24:58.063585 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:58.063555    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:58.064097 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:58.063843    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:24:58.064593 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:24:58.064573    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:25:10.057176 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.057144    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"]
Oct 31 03:25:10.061254 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.061232    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.156232 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.156189    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b59563\"/\"openshift-service-ca.crt\""
Oct 31 03:25:10.158032 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.158016    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b59563\"/\"kube-root-ca.crt\""
Oct 31 03:25:10.158128 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.158046    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b59563\"/\"default-dockercfg-jq9mp\""
Oct 31 03:25:10.158128 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.158101    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b59563\"/\"distinct-w2-b59563-kserve-self-signed-certs\""
Oct 31 03:25:10.160088 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.160071    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"]
Oct 31 03:25:10.166631 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.166606    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-model-cache\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.166718 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.166644    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-home\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.166718 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.166672    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q9s4\" (UniqueName: \"kubernetes.io/projected/8164484e-3721-4998-bb76-278d2f4b5e35-kube-api-access-6q9s4\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.166867 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.166844    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-dshm\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.166951 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.166920    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-tmp-dir\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.166993 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.166978    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8164484e-3721-4998-bb76-278d2f4b5e35-tls-certs\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.268675 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.268639    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-model-cache\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.268675 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.268690    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-home\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.268998 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.268723    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q9s4\" (UniqueName: \"kubernetes.io/projected/8164484e-3721-4998-bb76-278d2f4b5e35-kube-api-access-6q9s4\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.269158 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.269127    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-dshm\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.269158 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.269160    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-model-cache\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.269363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.269200    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-tmp-dir\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.269363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.269222    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-home\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.269363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.269272    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8164484e-3721-4998-bb76-278d2f4b5e35-tls-certs\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.269580 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.269561    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-tmp-dir\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.271612 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.271581    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-dshm\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.272179 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.272151    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8164484e-3721-4998-bb76-278d2f4b5e35-tls-certs\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.332363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.332283    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q9s4\" (UniqueName: \"kubernetes.io/projected/8164484e-3721-4998-bb76-278d2f4b5e35-kube-api-access-6q9s4\") pod \"distinct-w2-b59563-kserve-5f4d469db7-7w897\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.374358 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.374310    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:10.769003 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.768975    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"]
Oct 31 03:25:10.772077 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:25:10.772046    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8164484e_3721_4998_bb76_278d2f4b5e35.slice/crio-b7369da734a0800293fca9b28f69e7c5c4625b3992cfcfb10c0fbcf1b7148ab7 WatchSource:0}: Error finding container b7369da734a0800293fca9b28f69e7c5c4625b3992cfcfb10c0fbcf1b7148ab7: Status 404 returned error can't find the container with id b7369da734a0800293fca9b28f69e7c5c4625b3992cfcfb10c0fbcf1b7148ab7
Oct 31 03:25:10.894696 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.894654    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897" event={"ID":"8164484e-3721-4998-bb76-278d2f4b5e35","Type":"ContainerStarted","Data":"87e3d56414dbac717bd403826421dfb08daa52edc3d15e3c3c26f5f25ca19197"}
Oct 31 03:25:10.894977 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:10.894955    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897" event={"ID":"8164484e-3721-4998-bb76-278d2f4b5e35","Type":"ContainerStarted","Data":"b7369da734a0800293fca9b28f69e7c5c4625b3992cfcfb10c0fbcf1b7148ab7"}
Oct 31 03:25:11.007301 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:11.007241    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897" podStartSLOduration=1.007226479 podStartE2EDuration="1.007226479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:11.006128666 +0000 UTC m=+1052.566608398" watchObservedRunningTime="2026-10-31 03:25:11.007226479 +0000 UTC m=+1052.567706210"
Oct 31 03:25:11.374955 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:11.374919    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:16.175942 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:25:16.175900    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:25:20.375069 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:20.375026    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:20.388619 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:20.388585    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:20.957984 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:20.957956    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:25:22.881319 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:22.881281    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"]
Oct 31 03:25:22.885232 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:22.885201    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.010889 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.010838    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.011082 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.010961    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.011082 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.010996    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.011190 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.011095    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.011190 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.011126    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.011190 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.011170    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.011293 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.011254    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldprm\" (UniqueName: \"kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-kube-api-access-ldprm\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.011352 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.011337    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.011401 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.011387    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.086342 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.086302    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"]
Oct 31 03:25:23.112163 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.112127    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.112369 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.112180    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.112369 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.112222    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.112491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.112387    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.112491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.112434    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.112603 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.112510    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.112603 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.112539    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.112603 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.112570    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.112760 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.112656    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.112760 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.112656    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.112760 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.112657    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldprm\" (UniqueName: \"kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-kube-api-access-ldprm\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.112940 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.112919    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.113029 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.113001    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.113276 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.113254    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.115325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.115293    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.115452 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.115433    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.167064 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.166988    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.167241 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.167082    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldprm\" (UniqueName: \"kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-kube-api-access-ldprm\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.198595 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.198560    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:23.624834 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.624807    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"]
Oct 31 03:25:23.626950 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:25:23.626922    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d8e6e8_60c1_48ef_9630_dc37ab279bf7.slice/crio-eacc370b890b6e8bb09d867b2f3a3137b57724c6e4e1324671f4ab438620f560 WatchSource:0}: Error finding container eacc370b890b6e8bb09d867b2f3a3137b57724c6e4e1324671f4ab438620f560: Status 404 returned error can't find the container with id eacc370b890b6e8bb09d867b2f3a3137b57724c6e4e1324671f4ab438620f560
Oct 31 03:25:23.629356 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.629321    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:25:23.629446 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.629392    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:25:23.629446 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.629421    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:25:23.965848 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.965751    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" event={"ID":"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7","Type":"ContainerStarted","Data":"86cb37c0c6ca1ebb779f1c7d144e5531f613bea5d85f874ab89d48289413182f"}
Oct 31 03:25:23.965848 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:23.965807    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" event={"ID":"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7","Type":"ContainerStarted","Data":"eacc370b890b6e8bb09d867b2f3a3137b57724c6e4e1324671f4ab438620f560"}
Oct 31 03:25:24.078893 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:24.078810    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" podStartSLOduration=2.078789301 podStartE2EDuration="2.078789301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:24.077313185 +0000 UTC m=+1065.637792938" watchObservedRunningTime="2026-10-31 03:25:24.078789301 +0000 UTC m=+1065.639269032"
Oct 31 03:25:24.199317 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:24.199284    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:24.200812 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:24.200780    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:25:24.200987 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:24.200836    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" podUID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 03:25:25.199811 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:25.199760    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:25:25.200371 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:25.199818    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" podUID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 03:25:26.199594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:26.199553    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:25:26.199794 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:26.199613    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" podUID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 03:25:27.200038 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:27.200004    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:25:27.200443 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:27.200087    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" podUID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 03:25:28.198960 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:28.198925    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:25:28.199190 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:28.198982    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" podUID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 03:25:29.199225 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:29.199189    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:25:29.199633 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:29.199256    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" podUID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 03:25:30.199599 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:30.199551    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:25:30.200055 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:30.199639    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" podUID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 03:25:32.085794 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:32.085758    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:32.086731 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:32.086705    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:32.087652 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:32.087630    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:25:46.175487 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:25:46.175397    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:25:51.841957 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:51.841918    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"]
Oct 31 03:25:51.845845 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:51.845823    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:51.895572 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:51.895537    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b59563-openshift-default-dockercfg-h4q7c\""
Oct 31 03:25:51.949647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:51.949605    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"]
Oct 31 03:25:51.961845 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:51.961807    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:51.962067 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:51.961853    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:51.962067 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:51.961968    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:51.962194 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:51.962072    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:51.962194 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:51.962129    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:51.962194 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:51.962166    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78dr\" (UniqueName: \"kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-kube-api-access-m78dr\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:51.962351 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:51.962206    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc949ce1-87f6-403b-a72d-1a517f979a65-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:51.962351 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:51.962241    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:51.962351 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:51.962299    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.063866 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.063826    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.063866 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.063867    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.064146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.063915    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.064146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.063961    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.064146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.063995    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.064146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.064018    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78dr\" (UniqueName: \"kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-kube-api-access-m78dr\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.064146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.064056    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc949ce1-87f6-403b-a72d-1a517f979a65-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.064146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.064090    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.064146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.064133    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.064480 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.064324    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.064480 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.064382    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.064577 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.064478    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.064617 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.064581    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.064892 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.064853    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc949ce1-87f6-403b-a72d-1a517f979a65-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.066630 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.066604    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.066786 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.066768    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.119116 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.119043    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.119278 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.119252    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78dr\" (UniqueName: \"kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-kube-api-access-m78dr\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.159210 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.159177    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:52.582398 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.582369    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"]
Oct 31 03:25:52.585045 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:25:52.585014    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc949ce1_87f6_403b_a72d_1a517f979a65.slice/crio-3cd630d8bf5ba34bee3cd64667a6361f83b64446d3160507ba240daa4124aad7 WatchSource:0}: Error finding container 3cd630d8bf5ba34bee3cd64667a6361f83b64446d3160507ba240daa4124aad7: Status 404 returned error can't find the container with id 3cd630d8bf5ba34bee3cd64667a6361f83b64446d3160507ba240daa4124aad7
Oct 31 03:25:52.587519 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.587482    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:25:52.587634 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.587568    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:25:52.587634 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:52.587613    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:25:53.152193 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:53.152144    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" event={"ID":"dc949ce1-87f6-403b-a72d-1a517f979a65","Type":"ContainerStarted","Data":"a0a5738c01a7ace436549c325b7d092835038e77d9b77c90da25cf4503ff2ebe"}
Oct 31 03:25:53.152614 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:53.152262    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" event={"ID":"dc949ce1-87f6-403b-a72d-1a517f979a65","Type":"ContainerStarted","Data":"3cd630d8bf5ba34bee3cd64667a6361f83b64446d3160507ba240daa4124aad7"}
Oct 31 03:25:53.159646 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:53.159619    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:25:53.161054 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:53.161032    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 03:25:53.161167 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:53.161075    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" podUID="dc949ce1-87f6-403b-a72d-1a517f979a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 03:25:53.268545 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:53.268483    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" podStartSLOduration=2.268466651 podStartE2EDuration="2.268466651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:53.266286203 +0000 UTC m=+1094.826765935" watchObservedRunningTime="2026-10-31 03:25:53.268466651 +0000 UTC m=+1094.828946378"
Oct 31 03:25:54.168027 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:54.167987    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 03:25:54.170454 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:54.170417    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" podUID="dc949ce1-87f6-403b-a72d-1a517f979a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 03:25:55.170903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:55.169987    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 03:25:55.170903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:55.170053    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" podUID="dc949ce1-87f6-403b-a72d-1a517f979a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 03:25:56.168901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:56.168314    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 03:25:56.168901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:56.168385    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" podUID="dc949ce1-87f6-403b-a72d-1a517f979a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 03:25:57.168901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:57.168322    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 03:25:57.168901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:57.168398    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" podUID="dc949ce1-87f6-403b-a72d-1a517f979a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 03:25:58.165423 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:58.165378    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 03:25:58.165764 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:58.165728    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" podUID="dc949ce1-87f6-403b-a72d-1a517f979a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 03:25:59.160192 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:59.160140    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 03:25:59.160726 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:25:59.160217    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" podUID="dc949ce1-87f6-403b-a72d-1a517f979a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 03:26:00.164122 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:00.164091    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:26:00.164532 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:00.164354    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:26:00.165031 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:00.165016    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 03:26:03.786207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:03.786165    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"]
Oct 31 03:26:16.174404 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:26:16.174359    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:26:18.806949 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:18.806907    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg" podUID="20a3fef2-03e5-4fd0-86e1-1ae47cdfc642" containerName="main" containerID="cri-o://b70fda5a64b5c5f2540e7cb1a248dbf8b73e17feef2275967049a9408068c4d4" gracePeriod=45
Oct 31 03:26:19.083509 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.083478    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:26:19.103714 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.103678    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-model-cache\") pod \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") "
Oct 31 03:26:19.103938 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.103775    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-home\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-home\") pod \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") "
Oct 31 03:26:19.103938 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.103814    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tls-certs\" (UniqueName: \"kubernetes.io/secret/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tls-certs\") pod \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") "
Oct 31 03:26:19.103938 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.103914    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-kube-api-access-fx2gj\" (UniqueName: \"kubernetes.io/projected/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-kube-api-access-fx2gj\") pod \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") "
Oct 31 03:26:19.104129 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.103999    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-dshm\") pod \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") "
Oct 31 03:26:19.104129 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.104025    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tmp-dir\") pod \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\" (UID: \"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642\") "
Oct 31 03:26:19.104129 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.104056    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-model-cache" pod "20a3fef2-03e5-4fd0-86e1-1ae47cdfc642" (UID: "20a3fef2-03e5-4fd0-86e1-1ae47cdfc642"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:26:19.104129 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.104071    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-home" pod "20a3fef2-03e5-4fd0-86e1-1ae47cdfc642" (UID: "20a3fef2-03e5-4fd0-86e1-1ae47cdfc642"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:26:19.104377 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.104354    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tmp-dir" pod "20a3fef2-03e5-4fd0-86e1-1ae47cdfc642" (UID: "20a3fef2-03e5-4fd0-86e1-1ae47cdfc642"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:26:19.104445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.104430    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:19.104498 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.104447    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:19.106434 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.106414    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-dshm" pod "20a3fef2-03e5-4fd0-86e1-1ae47cdfc642" (UID: "20a3fef2-03e5-4fd0-86e1-1ae47cdfc642"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:26:19.106576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.106562    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tls-certs" pod "20a3fef2-03e5-4fd0-86e1-1ae47cdfc642" (UID: "20a3fef2-03e5-4fd0-86e1-1ae47cdfc642"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:26:19.106646 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.106622    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-kube-api-access-fx2gj" pod "20a3fef2-03e5-4fd0-86e1-1ae47cdfc642" (UID: "20a3fef2-03e5-4fd0-86e1-1ae47cdfc642"). InnerVolumeSpecName "kube-api-access-fx2gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:26:19.205925 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.205871    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:19.205925 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.205927    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:19.205925 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.205937    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:19.206220 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.205945    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx2gj\" (UniqueName: \"kubernetes.io/projected/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642-kube-api-access-fx2gj\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:19.318000 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.317914    5415 generic.go:356] "Generic (PLEG): container finished" podID="20a3fef2-03e5-4fd0-86e1-1ae47cdfc642" containerID="b70fda5a64b5c5f2540e7cb1a248dbf8b73e17feef2275967049a9408068c4d4" exitCode=0
Oct 31 03:26:19.318000 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.317953    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"
Oct 31 03:26:19.318000 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.317974    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg" event={"ID":"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642","Type":"ContainerDied","Data":"b70fda5a64b5c5f2540e7cb1a248dbf8b73e17feef2275967049a9408068c4d4"}
Oct 31 03:26:19.318223 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.318022    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg" event={"ID":"20a3fef2-03e5-4fd0-86e1-1ae47cdfc642","Type":"ContainerDied","Data":"502557bd1e23f6fcfa5d07830def5418c6281bb52627b6d4a113edae81a41afd"}
Oct 31 03:26:19.318223 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.318046    5415 scope.go:122] "RemoveContainer" containerID="b70fda5a64b5c5f2540e7cb1a248dbf8b73e17feef2275967049a9408068c4d4"
Oct 31 03:26:19.328366 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.328347    5415 scope.go:122] "RemoveContainer" containerID="b70fda5a64b5c5f2540e7cb1a248dbf8b73e17feef2275967049a9408068c4d4"
Oct 31 03:26:19.328676 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:26:19.328658    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70fda5a64b5c5f2540e7cb1a248dbf8b73e17feef2275967049a9408068c4d4\": container with ID starting with b70fda5a64b5c5f2540e7cb1a248dbf8b73e17feef2275967049a9408068c4d4 not found: ID does not exist" containerID="b70fda5a64b5c5f2540e7cb1a248dbf8b73e17feef2275967049a9408068c4d4"
Oct 31 03:26:19.328715 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.328689    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70fda5a64b5c5f2540e7cb1a248dbf8b73e17feef2275967049a9408068c4d4"} err="failed to get container status \"b70fda5a64b5c5f2540e7cb1a248dbf8b73e17feef2275967049a9408068c4d4\": rpc error: code = NotFound desc = could not find container \"b70fda5a64b5c5f2540e7cb1a248dbf8b73e17feef2275967049a9408068c4d4\": container with ID starting with b70fda5a64b5c5f2540e7cb1a248dbf8b73e17feef2275967049a9408068c4d4 not found: ID does not exist"
Oct 31 03:26:19.483142 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.483105    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"]
Oct 31 03:26:19.490183 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:19.490149    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/e2e-conflict-a-3b9043d1-kserve-bc846f5cc-5d6tg"]
Oct 31 03:26:20.606047 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:20.606014    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a3fef2-03e5-4fd0-86e1-1ae47cdfc642" path="/var/lib/kubelet/pods/20a3fef2-03e5-4fd0-86e1-1ae47cdfc642/volumes"
Oct 31 03:26:31.298605 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.298566    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"]
Oct 31 03:26:31.299125 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.299110    5415 cpu_manager.go:414] "containerMap: removing container" podUID="20a3fef2-03e5-4fd0-86e1-1ae47cdfc642" containerName="main"
Oct 31 03:26:31.299170 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.299128    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a3fef2-03e5-4fd0-86e1-1ae47cdfc642" containerName="main"
Oct 31 03:26:31.299235 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.299226    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a3fef2-03e5-4fd0-86e1-1ae47cdfc642" containerName="main"
Oct 31 03:26:31.302647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.302622    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.389405 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.389362    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.389616 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.389432    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.389616 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.389471    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.389746 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.389608    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9lmn\" (UniqueName: \"kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-kube-api-access-p9lmn\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.389746 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.389644    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.389746 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.389687    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37159aa5-167a-4b63-a39c-86a341139168-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.390011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.389981    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.390150 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.390067    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37159aa5-167a-4b63-a39c-86a341139168-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.390150 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.390126    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.404251 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.404214    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"]
Oct 31 03:26:31.490801 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.490768    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.490801 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.490809    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.491084 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.490846    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9lmn\" (UniqueName: \"kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-kube-api-access-p9lmn\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.491084 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.490867    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.491084 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.490910    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37159aa5-167a-4b63-a39c-86a341139168-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.491084 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.491049    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.491084 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.491077    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37159aa5-167a-4b63-a39c-86a341139168-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.491362 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.491114    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.491362 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.491156    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.491362 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.491245    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.491362 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.491302    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.491599 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.491519    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.491599 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.491578    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.492024 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.491998    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37159aa5-167a-4b63-a39c-86a341139168-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.493196 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.493175    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.493418 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.493398    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37159aa5-167a-4b63-a39c-86a341139168-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.546241 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.546205    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.546412 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.546287    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9lmn\" (UniqueName: \"kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-kube-api-access-p9lmn\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:31.615850 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:31.615814    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:32.045659 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:32.045635    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"]
Oct 31 03:26:32.047341 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:26:32.047312    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37159aa5_167a_4b63_a39c_86a341139168.slice/crio-006ff97c74eb8d1ba9c99f101317046fb427e9f7c0c4d5c60341d20dd828dada WatchSource:0}: Error finding container 006ff97c74eb8d1ba9c99f101317046fb427e9f7c0c4d5c60341d20dd828dada: Status 404 returned error can't find the container with id 006ff97c74eb8d1ba9c99f101317046fb427e9f7c0c4d5c60341d20dd828dada
Oct 31 03:26:32.049788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:32.049754    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:26:32.049922 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:32.049824    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:26:32.049922 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:32.049855    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:26:32.412356 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:32.412306    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r" event={"ID":"37159aa5-167a-4b63-a39c-86a341139168","Type":"ContainerStarted","Data":"e6afe4588d53ebaf27211854ca57f0a360c0853cf11aedae3f3ab2cd0cfe76f1"}
Oct 31 03:26:32.412804 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:32.412379    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r" event={"ID":"37159aa5-167a-4b63-a39c-86a341139168","Type":"ContainerStarted","Data":"006ff97c74eb8d1ba9c99f101317046fb427e9f7c0c4d5c60341d20dd828dada"}
Oct 31 03:26:32.526182 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:32.526126    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r" podStartSLOduration=1.526110031 podStartE2EDuration="1.526110031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:32.525196662 +0000 UTC m=+1134.085676393" watchObservedRunningTime="2026-10-31 03:26:32.526110031 +0000 UTC m=+1134.086589762"
Oct 31 03:26:32.616410 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:32.616374    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:33.618323 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:33.618282    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 03:26:33.618748 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:33.618355    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r" podUID="37159aa5-167a-4b63-a39c-86a341139168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 03:26:34.616592 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:34.616555    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 03:26:34.616815 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:34.616626    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r" podUID="37159aa5-167a-4b63-a39c-86a341139168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 03:26:35.616700 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:35.616576    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 03:26:35.616700 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:35.616633    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r" podUID="37159aa5-167a-4b63-a39c-86a341139168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 03:26:36.617026 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:36.616989    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 03:26:36.617454 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:36.617067    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r" podUID="37159aa5-167a-4b63-a39c-86a341139168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 03:26:37.616983 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:37.616947    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 03:26:37.617198 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:37.617013    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r" podUID="37159aa5-167a-4b63-a39c-86a341139168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 03:26:38.490269 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.490234    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"]
Oct 31 03:26:38.504201 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.504155    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.592082 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.592048    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"]
Oct 31 03:26:38.617110 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.617073    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 03:26:38.617282 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.617178    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r" podUID="37159aa5-167a-4b63-a39c-86a341139168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 03:26:38.702978 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.702935    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.703182 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.703003    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.703182 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.703104    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.703182 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.703136    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.703363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.703187    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxnwf\" (UniqueName: \"kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-kube-api-access-vxnwf\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.703363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.703289    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.703482 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.703362    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.703482 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.703461    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3173f7-3b7d-4765-94b9-506ca9765abf-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.703585 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.703500    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.804254 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.804207    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxnwf\" (UniqueName: \"kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-kube-api-access-vxnwf\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.804468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.804274    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.804468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.804327    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.804468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.804451    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3173f7-3b7d-4765-94b9-506ca9765abf-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.804632 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.804494    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.804632 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.804580    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.804735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.804668    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.804735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.804697    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.804735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.804721    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.804913 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.804819    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.804996 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.804969    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.805103 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.805081    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.805180 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.805148    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.805312 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.805283    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3173f7-3b7d-4765-94b9-506ca9765abf-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.806982 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.806952    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.807127 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.807112    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.859444 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.859408    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:38.859609 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:38.859527    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxnwf\" (UniqueName: \"kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-kube-api-access-vxnwf\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:39.122127 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:39.122082    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:39.576266 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:39.576184    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"]
Oct 31 03:26:39.605490 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:26:39.605450    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3173f7_3b7d_4765_94b9_506ca9765abf.slice/crio-4b4d87fe749dea0a54794e56675d58f68f23e49888bbe728ac2b7ac2666cec9d WatchSource:0}: Error finding container 4b4d87fe749dea0a54794e56675d58f68f23e49888bbe728ac2b7ac2666cec9d: Status 404 returned error can't find the container with id 4b4d87fe749dea0a54794e56675d58f68f23e49888bbe728ac2b7ac2666cec9d
Oct 31 03:26:39.608136 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:39.608081    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:26:39.608248 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:39.608185    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:26:39.608248 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:39.608226    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:26:39.621751 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:39.621726    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:39.622097 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:39.622060    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:39.622985 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:39.622966    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:26:40.473803 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:40.473767    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr" event={"ID":"0f3173f7-3b7d-4765-94b9-506ca9765abf","Type":"ContainerStarted","Data":"b776103e4294dcf23bee9f88057592a8117ad30ae534eb0d3a79d8c36c27717f"}
Oct 31 03:26:40.474169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:40.474147    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr" event={"ID":"0f3173f7-3b7d-4765-94b9-506ca9765abf","Type":"ContainerStarted","Data":"4b4d87fe749dea0a54794e56675d58f68f23e49888bbe728ac2b7ac2666cec9d"}
Oct 31 03:26:40.602938 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:40.593387    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr" podStartSLOduration=2.593358542 podStartE2EDuration="2.593358542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:40.593152761 +0000 UTC m=+1142.153632492" watchObservedRunningTime="2026-10-31 03:26:40.593358542 +0000 UTC m=+1142.153838275"
Oct 31 03:26:41.123337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:41.123251    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:41.132403 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:41.132036    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 03:26:41.132403 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:41.132102    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr" podUID="0f3173f7-3b7d-4765-94b9-506ca9765abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 03:26:42.123434 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:42.123387    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 03:26:42.124608 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:42.124138    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr" podUID="0f3173f7-3b7d-4765-94b9-506ca9765abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 03:26:42.285915 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:42.281009    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt" podUID="7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:26:42.402228 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:42.402120    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" podUID="7286f18c-2456-47f7-a04d-078677b37a27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 31 03:26:42.690765 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:42.689999    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:26:43.123553 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:43.123507    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 03:26:43.124049 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:43.123572    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr" podUID="0f3173f7-3b7d-4765-94b9-506ca9765abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 03:26:44.122764 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:44.122727    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 03:26:44.122982 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:44.122783    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr" podUID="0f3173f7-3b7d-4765-94b9-506ca9765abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 03:26:45.123303 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:45.123260    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 03:26:45.123707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:45.123337    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr" podUID="0f3173f7-3b7d-4765-94b9-506ca9765abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 03:26:46.123569 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:46.123532    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 03:26:46.123983 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:46.123595    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr" podUID="0f3173f7-3b7d-4765-94b9-506ca9765abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 03:26:46.199167 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:26:46.199131    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:26:47.123373 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:47.123330    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 03:26:47.123563 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:47.123401    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr" podUID="0f3173f7-3b7d-4765-94b9-506ca9765abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 03:26:48.658043 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:48.658008    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:48.658480 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:48.658403    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:26:48.659439 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:26:48.659418    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:27:01.543688 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.543654    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"]
Oct 31 03:27:01.547730 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.547711    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.724220 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.724180    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.724422 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.724265    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.724422 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.724286    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.724544 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.724442    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b328fb-b191-4e90-92bf-671dae7a3f89-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.724544 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.724496    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.724634 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.724557    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.724682 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.724630    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.724682 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.724675    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.724815 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.724781    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8cc\" (UniqueName: \"kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-kube-api-access-5v8cc\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.749001 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.748936    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"]
Oct 31 03:27:01.826238 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.826143    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.826238 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.826209    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b328fb-b191-4e90-92bf-671dae7a3f89-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.826238 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.826234    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.826534 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.826254    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.826534 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.826305    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.826534 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.826341    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.826534 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.826362    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8cc\" (UniqueName: \"kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-kube-api-access-5v8cc\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.826534 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.826432    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.826534 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.826520    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.826863 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.826768    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.826969 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.826861    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.827027 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.826978    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.827090 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.827051    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.827388 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.827365    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b328fb-b191-4e90-92bf-671dae7a3f89-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.828726 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.828700    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.828931 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.828912    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.882178 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.882132    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:01.882339 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:01.882186    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8cc\" (UniqueName: \"kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-kube-api-access-5v8cc\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:02.162198 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:02.162158    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:02.585760 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:02.585726    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"]
Oct 31 03:27:02.623568 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:27:02.623535    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b328fb_b191_4e90_92bf_671dae7a3f89.slice/crio-b4150f31035518288d2cd7bfaca8e7e21a6d77c3024e56e8ca5fd8672e7fb953 WatchSource:0}: Error finding container b4150f31035518288d2cd7bfaca8e7e21a6d77c3024e56e8ca5fd8672e7fb953: Status 404 returned error can't find the container with id b4150f31035518288d2cd7bfaca8e7e21a6d77c3024e56e8ca5fd8672e7fb953
Oct 31 03:27:02.626134 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:02.626103    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:27:02.626229 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:02.626173    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:27:02.626229 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:02.626209    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:27:02.695220 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:02.695194    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274" event={"ID":"55b328fb-b191-4e90-92bf-671dae7a3f89","Type":"ContainerStarted","Data":"b4150f31035518288d2cd7bfaca8e7e21a6d77c3024e56e8ca5fd8672e7fb953"}
Oct 31 03:27:03.711776 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:03.711734    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274" event={"ID":"55b328fb-b191-4e90-92bf-671dae7a3f89","Type":"ContainerStarted","Data":"7757c20bc7ccede8a9a5bbcc559b2eecc35c7062f3ad3459a9047706929fb3e8"}
Oct 31 03:27:04.170340 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:04.170301    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:04.172935 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:04.172898    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 03:27:04.173212 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:04.173181    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274" podUID="55b328fb-b191-4e90-92bf-671dae7a3f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 03:27:05.203903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:05.196276    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 03:27:05.235119 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:05.235061    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274" podUID="55b328fb-b191-4e90-92bf-671dae7a3f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 03:27:05.494963 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:05.494030    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" podUID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:05.649081 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:05.649018    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:06.167903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:06.167241    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 03:27:06.167903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:06.167307    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274" podUID="55b328fb-b191-4e90-92bf-671dae7a3f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 03:27:07.168901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.168009    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 03:27:07.168901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.168079    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274" podUID="55b328fb-b191-4e90-92bf-671dae7a3f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 03:27:07.481198 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.481068    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274" podStartSLOduration=6.481052773 podStartE2EDuration="6.481052773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:03.927662442 +0000 UTC m=+1165.488142217" watchObservedRunningTime="2026-10-31 03:27:07.481052773 +0000 UTC m=+1169.041532504"
Oct 31 03:27:07.482381 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.482340    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"]
Oct 31 03:27:07.487237 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.487212    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.585923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.581859    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"]
Oct 31 03:27:07.585923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.584730    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.585923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.585054    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.585923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.585111    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.585923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.585191    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.585923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.585279    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.585923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.585302    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.585923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.585371    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glz67\" (UniqueName: \"kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-kube-api-access-glz67\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.585923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.585399    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.585923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.585415    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.687097 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.687057    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.687097 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.687102    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.687376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.687151    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.687376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.687178    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.687376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.687225    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.687376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.687248    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.687376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.687344    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glz67\" (UniqueName: \"kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-kube-api-access-glz67\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.687640 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.687384    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.687640 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.687408    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.687640 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.687625    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.687961 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.687936    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.688072 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.688044    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.688120 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.688089    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.688238 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.688218    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.690130 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.690105    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.690220 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.690203    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.745812 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.745719    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.745988 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.745832    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glz67\" (UniqueName: \"kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-kube-api-access-glz67\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:07.803599 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:07.803547    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:08.172078 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:08.172036    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 03:27:08.172582 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:08.172100    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274" podUID="55b328fb-b191-4e90-92bf-671dae7a3f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 03:27:08.204654 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:08.204626    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"]
Oct 31 03:27:08.207131 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:27:08.207100    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae6e719_0fac_495f_b792_aecd2e6fd7cc.slice/crio-a84fccc060dfb35a67f15af6c27bef141214ca3391457d03f79761e489d8e522 WatchSource:0}: Error finding container a84fccc060dfb35a67f15af6c27bef141214ca3391457d03f79761e489d8e522: Status 404 returned error can't find the container with id a84fccc060dfb35a67f15af6c27bef141214ca3391457d03f79761e489d8e522
Oct 31 03:27:08.209589 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:08.209554    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:27:08.209706 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:08.209637    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:27:08.209706 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:08.209682    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:27:08.243285 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:08.243250    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn" event={"ID":"8ae6e719-0fac-495f-b792-aecd2e6fd7cc","Type":"ContainerStarted","Data":"a84fccc060dfb35a67f15af6c27bef141214ca3391457d03f79761e489d8e522"}
Oct 31 03:27:09.166900 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:09.164368    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 03:27:09.166900 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:09.164434    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274" podUID="55b328fb-b191-4e90-92bf-671dae7a3f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 03:27:09.293551 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:09.293509    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn" event={"ID":"8ae6e719-0fac-495f-b792-aecd2e6fd7cc","Type":"ContainerStarted","Data":"7754c8fe9636c45be48540d220378291dd092555078b713c2c7e41cef35263ba"}
Oct 31 03:27:09.481702 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:09.481569    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn" podStartSLOduration=2.481541299 podStartE2EDuration="2.481541299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:09.481295219 +0000 UTC m=+1171.041774955" watchObservedRunningTime="2026-10-31 03:27:09.481541299 +0000 UTC m=+1171.042021031"
Oct 31 03:27:09.804038 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:09.803942    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:09.805829 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:09.805798    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 03:27:09.805983 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:09.805858    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn" podUID="8ae6e719-0fac-495f-b792-aecd2e6fd7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 03:27:10.164395 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:10.164351    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 03:27:10.164728 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:10.164685    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274" podUID="55b328fb-b191-4e90-92bf-671dae7a3f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 03:27:10.804426 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:10.804386    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 03:27:10.804961 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:10.804443    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn" podUID="8ae6e719-0fac-495f-b792-aecd2e6fd7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 03:27:11.804518 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:11.804167    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 03:27:11.804518 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:11.804219    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn" podUID="8ae6e719-0fac-495f-b792-aecd2e6fd7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 03:27:12.096633 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:12.096598    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:12.096937 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:12.096914    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:12.097705 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:12.097685    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:27:12.804865 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:12.804821    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 03:27:12.805414 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:12.804927    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn" podUID="8ae6e719-0fac-495f-b792-aecd2e6fd7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 03:27:13.804237 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:13.804200    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 03:27:13.804419 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:13.804267    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn" podUID="8ae6e719-0fac-495f-b792-aecd2e6fd7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 03:27:14.804815 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:14.804766    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 03:27:14.805400 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:14.804827    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn" podUID="8ae6e719-0fac-495f-b792-aecd2e6fd7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 03:27:15.969601 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:15.969369    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:15.970394 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:15.970367    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:15.971077 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:15.971057    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:27:16.177831 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:27:16.177792    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:27:28.136906 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:28.133006    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd" podUID="e8a80930-06e4-4b4e-9aec-79274f0961f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.287905 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:28.287014    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt" podUID="7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.384903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:28.378996    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" podUID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 03:27:28.404038 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:28.403136    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" podUID="7286f18c-2456-47f7-a04d-078677b37a27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 31 03:27:28.623466 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:28.623080    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 31 03:27:29.156904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:29.133633    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd" podUID="e8a80930-06e4-4b4e-9aec-79274f0961f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.156904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:29.146006    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:27:29.156904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:29.152983    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:29.156904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:29.153045    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.314902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:29.314218    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt" podUID="7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.326910 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:29.318847    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:27:29.382079 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:29.382030    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" podUID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.382751 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:29.382724    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:27:29.418973 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:29.410744    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" podUID="7286f18c-2456-47f7-a04d-078677b37a27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.449916 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:29.419497    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:27:29.678814 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:29.674941    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 31 03:27:29.678814 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:29.678503    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:27:29.947376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:29.947272    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897" podUID="8164484e-3721-4998-bb76-278d2f4b5e35" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 31 03:27:30.136056 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:30.136006    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd" podUID="e8a80930-06e4-4b4e-9aec-79274f0961f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 31 03:27:30.323484 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:30.323378    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt" podUID="7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.404903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:30.395550    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" podUID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.448904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:30.439359    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" podUID="7286f18c-2456-47f7-a04d-078677b37a27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.686194 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:30.686149    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 31 03:27:30.954961 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:30.954714    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897" podUID="8164484e-3721-4998-bb76-278d2f4b5e35" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.955153 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:30.955108    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:27:31.163047 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:31.163004    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd" podUID="e8a80930-06e4-4b4e-9aec-79274f0961f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 31 03:27:31.231090 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:31.230988    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:31.231361 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:31.231329    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" podUID="dc949ce1-87f6-403b-a72d-1a517f979a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:31.256507 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:31.256475    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:27:31.258419 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:31.258397    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:27:31.296048 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:31.296018    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:27:31.336402 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:31.336369    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:27:31.399042 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:31.398992    5415 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897" podUID="8164484e-3721-4998-bb76-278d2f4b5e35" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:31.399956 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:31.399509    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" podUID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:31.458040 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:31.457991    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" podUID="7286f18c-2456-47f7-a04d-078677b37a27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:31.493165 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:31.493067    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:27:31.529440 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:31.529410    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:27:35.196056 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.196010    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"]
Oct 31 03:27:35.201512 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.201485    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.254304 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.254273    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-dshm\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.254492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.254334    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tls-certs\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.254492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.254440    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-home\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.254639 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.254625    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-model-cache\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.254679 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.254657    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tmp-dir\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.254713 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.254676    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z6pk\" (UniqueName: \"kubernetes.io/projected/f39ffc5b-0fbf-41ff-91c9-9242538ed564-kube-api-access-8z6pk\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.298738 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.298703    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"distinct-2-w1-d39257-kserve-self-signed-certs\""
Oct 31 03:27:35.303099 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.303062    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"]
Oct 31 03:27:35.355539 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.355498    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-model-cache\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.355539 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.355545    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tmp-dir\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.355805 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.355565    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z6pk\" (UniqueName: \"kubernetes.io/projected/f39ffc5b-0fbf-41ff-91c9-9242538ed564-kube-api-access-8z6pk\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.355805 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.355591    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-dshm\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.355805 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.355629    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tls-certs\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.355805 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.355654    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-home\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.356044 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.356027    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-model-cache\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.356101 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.356043    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tmp-dir\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.356142 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.356112    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-home\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.357996 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.357974    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-dshm\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.358199 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.358177    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tls-certs\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.412294 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.412252    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z6pk\" (UniqueName: \"kubernetes.io/projected/f39ffc5b-0fbf-41ff-91c9-9242538ed564-kube-api-access-8z6pk\") pod \"distinct-2-w1-d39257-kserve-648986f497-zrkdm\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.521927 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.521830    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:35.910406 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.910173    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"]
Oct 31 03:27:35.914371 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:27:35.914329    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf39ffc5b_0fbf_41ff_91c9_9242538ed564.slice/crio-1e736083b4c58b5b805e04f84e608764863496cc229e6b6b6f3d0c538e54209d WatchSource:0}: Error finding container 1e736083b4c58b5b805e04f84e608764863496cc229e6b6b6f3d0c538e54209d: Status 404 returned error can't find the container with id 1e736083b4c58b5b805e04f84e608764863496cc229e6b6b6f3d0c538e54209d
Oct 31 03:27:35.954547 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:35.954507    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm" event={"ID":"f39ffc5b-0fbf-41ff-91c9-9242538ed564","Type":"ContainerStarted","Data":"1e736083b4c58b5b805e04f84e608764863496cc229e6b6b6f3d0c538e54209d"}
Oct 31 03:27:36.963018 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:36.962974    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm" event={"ID":"f39ffc5b-0fbf-41ff-91c9-9242538ed564","Type":"ContainerStarted","Data":"f76f08e1fac660c6c5dbe8288e0be6adc22d071a5be4e0ced54bdcce506528c8"}
Oct 31 03:27:37.075732 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:37.075674    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm" podStartSLOduration=2.075659215 podStartE2EDuration="2.075659215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:37.073831671 +0000 UTC m=+1198.634311404" watchObservedRunningTime="2026-10-31 03:27:37.075659215 +0000 UTC m=+1198.636138945"
Oct 31 03:27:37.522525 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:37.522484    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:45.525900 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:45.524002    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:45.550405 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:45.549751    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:46.029034 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:46.029002    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:46.177230 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:27:46.177162    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:27:52.515201 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.515162    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"]
Oct 31 03:27:52.519166 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.519144    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.627450 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.627413    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"]
Oct 31 03:27:52.680044 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.679784    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.680190 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.680060    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbgbr\" (UniqueName: \"kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-kube-api-access-zbgbr\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.680190 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.680097    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.680284 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.680260    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.680320 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.680290    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.680353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.680318    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.680404 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.680352    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4743042-c4eb-407c-a8c1-6421259d5a3e-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.680536 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.680519    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.680576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.680553    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.782274 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.782172    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.782274 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.782215    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.782274 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.782255    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.782571 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.782301    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4743042-c4eb-407c-a8c1-6421259d5a3e-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.782571 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.782353    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.782571 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.782384    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.782571 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.782478    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.782571 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.782513    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbgbr\" (UniqueName: \"kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-kube-api-access-zbgbr\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.782571 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.782563    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.782869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.782658    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.782999 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.782972    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.783314 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.783286    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.783608 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.783585    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.784031 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.783983    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4743042-c4eb-407c-a8c1-6421259d5a3e-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.787390 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.787363    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.789389 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.789363    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.837382 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.837353    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:52.837550 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:52.837513    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbgbr\" (UniqueName: \"kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-kube-api-access-zbgbr\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:53.132396 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:53.132355    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:53.566764 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:53.566700    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"]
Oct 31 03:27:53.570500 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:27:53.570436    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4743042_c4eb_407c_a8c1_6421259d5a3e.slice/crio-5a8cedd59c00b3e8340967228527d571f83da411b037b7699ff5fc2e7812a2a9 WatchSource:0}: Error finding container 5a8cedd59c00b3e8340967228527d571f83da411b037b7699ff5fc2e7812a2a9: Status 404 returned error can't find the container with id 5a8cedd59c00b3e8340967228527d571f83da411b037b7699ff5fc2e7812a2a9
Oct 31 03:27:53.575050 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:53.575011    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:27:53.575174 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:53.575108    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:27:53.575174 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:53.575159    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:27:54.082811 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:54.082768    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" event={"ID":"d4743042-c4eb-407c-a8c1-6421259d5a3e","Type":"ContainerStarted","Data":"c9d909e102a83c70db3d8d24ad24acdfe02668d535841ccdb7dbc8e29f33b8ac"}
Oct 31 03:27:54.084019 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:54.083984    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" event={"ID":"d4743042-c4eb-407c-a8c1-6421259d5a3e","Type":"ContainerStarted","Data":"5a8cedd59c00b3e8340967228527d571f83da411b037b7699ff5fc2e7812a2a9"}
Oct 31 03:27:54.133397 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:54.133364    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:27:54.143073 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:54.143031    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 03:27:54.143381 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:54.143347    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 03:27:54.301949 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:54.301740    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" podStartSLOduration=2.301719025 podStartE2EDuration="2.301719025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:54.252393482 +0000 UTC m=+1215.812873238" watchObservedRunningTime="2026-10-31 03:27:54.301719025 +0000 UTC m=+1215.862198760"
Oct 31 03:27:55.138154 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:55.138117    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 03:27:55.161004 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:55.160954    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 03:27:55.316903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:55.307984    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:55.323067 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:55.323014    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd" podUID="e8a80930-06e4-4b4e-9aec-79274f0961f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:55.343900 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:55.342968    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897" podUID="8164484e-3721-4998-bb76-278d2f4b5e35" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 31 03:27:55.344108 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:55.343929    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt" podUID="7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:55.402486 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:55.402368    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" podUID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:55.540558 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:55.540512    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" podUID="7286f18c-2456-47f7-a04d-078677b37a27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.042230 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.042137    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm" podUID="f39ffc5b-0fbf-41ff-91c9-9242538ed564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.311010 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.218257    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 03:27:56.311010 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.218321    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 03:27:56.311756 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.311707    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.586900 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.586850    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:27:56.611492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.346977    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt" podUID="7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:27:56.611492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.347044    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd" podUID="e8a80930-06e4-4b4e-9aec-79274f0961f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 31 03:27:56.611492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.400595    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897" podUID="8164484e-3721-4998-bb76-278d2f4b5e35" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.611492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.444017    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" podUID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.611492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.541861    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" podUID="7286f18c-2456-47f7-a04d-078677b37a27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.611492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.566966    5415 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm" podUID="f39ffc5b-0fbf-41ff-91c9-9242538ed564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.611492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.588463    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:27:56.611492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.588512    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:27:56.611492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.588548    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:27:56.611492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.588586    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:27:56.611492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.588621    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:27:56.611492 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:56.597591    5415 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:57.070991 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.048696    5415 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" podUID="7286f18c-2456-47f7-a04d-078677b37a27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Oct 31 03:27:57.292990 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.091239    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm" podUID="f39ffc5b-0fbf-41ff-91c9-9242538ed564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:57.293924 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.148290    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 03:27:57.294129 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.294100    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 03:27:57.294321 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.294304    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:57.386805 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.386774    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 03:27:57.515393 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.515356    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 03:27:57.516636 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.516615    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 03:27:57.588262 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.588214    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:57.614209 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.614164    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt" podUID="7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:57.620522 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.620483    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" podUID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 03:27:57.644706 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.626427    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 03:27:57.644706 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.622557    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" podUID="7286f18c-2456-47f7-a04d-078677b37a27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 31 03:27:57.939771 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.939657    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 03:27:57.940185 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:57.940164    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 03:27:58.142155 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:58.142120    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 03:27:58.142430 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:58.142406    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 03:27:58.237697 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:58.237437    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 03:27:59.133435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:59.133403    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 03:27:59.133835 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:27:59.133458    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 03:28:00.132938 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:00.132866    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 03:28:00.133152 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:00.132963    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 03:28:01.133587 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:01.133552    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 03:28:01.134018 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:01.133606    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 03:28:02.133249 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:02.133207    5415 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 03:28:02.133412 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:02.133280    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 03:28:03.138294 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:03.138252    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:28:03.138852 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:03.138674    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:28:03.139452 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:03.139435    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:28:14.163181 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.163144    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"]
Oct 31 03:28:14.168351 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.168325    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:14.218964 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.218926    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6b3a29ec-serving-cert\""
Oct 31 03:28:14.248570 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.248532    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0c59d3-274c-4ecf-a8de-453ce7b43012-openshift-service-ca\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:14.248774 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.248594    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-maas-api-tls\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:14.248774 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.248697    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-metrics-certs\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:14.248774 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.248720    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mdk4\" (UniqueName: \"kubernetes.io/projected/bc0c59d3-274c-4ecf-a8de-453ce7b43012-kube-api-access-9mdk4\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:14.268908 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.268838    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"]
Oct 31 03:28:14.355299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.351261    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-metrics-certs\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:14.355299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.351330    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mdk4\" (UniqueName: \"kubernetes.io/projected/bc0c59d3-274c-4ecf-a8de-453ce7b43012-kube-api-access-9mdk4\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:14.355299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.351537    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0c59d3-274c-4ecf-a8de-453ce7b43012-openshift-service-ca\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:14.355299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.351587    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-maas-api-tls\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:14.355299 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:28:14.351829    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-serving-cert: secret "maas-api-e2e-mt-6b3a29ec-serving-cert" not found
Oct 31 03:28:14.355299 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:28:14.351944    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-maas-api-tls podName:bc0c59d3-274c-4ecf-a8de-453ce7b43012 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:14.851920625 +0000 UTC m=+1236.412400358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-maas-api-tls") pod "maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx" (UID: "bc0c59d3-274c-4ecf-a8de-453ce7b43012") : secret "maas-api-e2e-mt-6b3a29ec-serving-cert" not found
Oct 31 03:28:14.355299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.354238    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0c59d3-274c-4ecf-a8de-453ce7b43012-openshift-service-ca\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:14.359286 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.359235    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-metrics-certs\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:14.416774 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.416659    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mdk4\" (UniqueName: \"kubernetes.io/projected/bc0c59d3-274c-4ecf-a8de-453ce7b43012-kube-api-access-9mdk4\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:14.858645 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.858599    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-maas-api-tls\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:14.861208 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:14.861172    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-maas-api-tls\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:15.082346 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:15.082304    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:15.617605 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:15.617444    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"]
Oct 31 03:28:15.620577 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:28:15.620542    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0c59d3_274c_4ecf_a8de_453ce7b43012.slice/crio-c1b3e2890566e8ffa73de38cb4392f08bca09bdf4e3c9bffb8f80d3041cc14df WatchSource:0}: Error finding container c1b3e2890566e8ffa73de38cb4392f08bca09bdf4e3c9bffb8f80d3041cc14df: Status 404 returned error can't find the container with id c1b3e2890566e8ffa73de38cb4392f08bca09bdf4e3c9bffb8f80d3041cc14df
Oct 31 03:28:15.622448 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:15.622427    5415 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:28:15.721630 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:15.721589    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx" event={"ID":"bc0c59d3-274c-4ecf-a8de-453ce7b43012","Type":"ContainerStarted","Data":"c1b3e2890566e8ffa73de38cb4392f08bca09bdf4e3c9bffb8f80d3041cc14df"}
Oct 31 03:28:16.240398 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:28:16.240298    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:28:17.806690 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:17.806649    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx" event={"ID":"bc0c59d3-274c-4ecf-a8de-453ce7b43012","Type":"ContainerStarted","Data":"cf4ea7cad25b06fa45e538655d28c5e9ede0154ae0dd00dd7330d4151cd93c62"}
Oct 31 03:28:17.807214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:17.806857    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:17.919423 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:17.919350    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx" podStartSLOduration=2.816581114 podStartE2EDuration="3.919329342s" totalImagesPullingTime="1.102748228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:17.916734152 +0000 UTC m=+1239.477213884" watchObservedRunningTime="2026-10-31 03:28:17.919329342 +0000 UTC m=+1239.479809073"
Oct 31 03:28:23.818479 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:28:23.818444    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:28:46.175280 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:28:46.175187    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:29:03.851212 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:03.851176    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"]
Oct 31 03:29:03.855695 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:03.855668    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:03.932214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:03.932180    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"]
Oct 31 03:29:03.976799 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:03.976764    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd5b85\"/\"distinct-w3-dd5b85-kserve-self-signed-certs\""
Oct 31 03:29:04.016537 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.016498    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-dshm\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.016723 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.016661    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tmp-dir\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.016799 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.016783    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tls-certs\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.016850 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.016819    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-model-cache\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.016850 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.016837    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgw8v\" (UniqueName: \"kubernetes.io/projected/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-kube-api-access-bgw8v\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.016967 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.016868    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-home\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.118454 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.118361    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tls-certs\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.118454 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.118432    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-model-cache\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.118695 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.118465    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgw8v\" (UniqueName: \"kubernetes.io/projected/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-kube-api-access-bgw8v\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.118695 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.118515    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-home\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.118695 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.118597    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-dshm\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.118898 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.118707    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tmp-dir\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.118961 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.118913    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-model-cache\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.119138 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.119114    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tmp-dir\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.119296 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.119263    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-home\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.125181 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.123004    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-dshm\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.125181 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.123453    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tls-certs\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.173980 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.173942    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgw8v\" (UniqueName: \"kubernetes.io/projected/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-kube-api-access-bgw8v\") pod \"distinct-w3-dd5b85-kserve-69c9d8b874-k92xl\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.468739 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.468651    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:04.894256 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:04.889934    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"]
Oct 31 03:29:05.154136 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:05.154009    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl" event={"ID":"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51","Type":"ContainerStarted","Data":"055c65178450b786580a01920c713df5fbc12a8aad7463d0b20e258a937d16b5"}
Oct 31 03:29:05.155523 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:05.155485    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl" event={"ID":"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51","Type":"ContainerStarted","Data":"a822223496e9d6027a3479ad4e986c75e9e7679e8f63c1199411197cfa4a9345"}
Oct 31 03:29:05.266785 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:05.266717    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl" podStartSLOduration=2.266698724 podStartE2EDuration="2.266698724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:05.265169996 +0000 UTC m=+1286.825649728" watchObservedRunningTime="2026-10-31 03:29:05.266698724 +0000 UTC m=+1286.827178433"
Oct 31 03:29:05.469973 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:05.469463    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:14.469692 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:14.469656    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:14.482359 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:14.482331    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:15.266002 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:15.265973    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 03:29:16.176952 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:29:16.176909    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:29:31.353242 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.353205    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"]
Oct 31 03:29:31.357483 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.357454    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.390569 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.390525    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srnh\" (UniqueName: \"kubernetes.io/projected/3947643c-111e-4350-9749-47643517df5c-kube-api-access-8srnh\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.390569 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.390573    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-home\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.390833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.390595    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-tmp-dir\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.390931 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.390837    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-model-cache\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.390998 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.390958    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-dshm\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.391103 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.391087    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3947643c-111e-4350-9749-47643517df5c-tls-certs\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.453723 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.453691    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6b3a29ec\"/\"openshift-service-ca.crt\""
Oct 31 03:29:31.453958 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.453742    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6b3a29ec\"/\"kube-root-ca.crt\""
Oct 31 03:29:31.453958 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.453905    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6b3a29ec\"/\"default-dockercfg-pnz8c\""
Oct 31 03:29:31.454070 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.453988    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6b3a29ec\"/\"e2e-lifecycle-model-6b3a29ec-kserve-self-signed-certs\""
Oct 31 03:29:31.458349 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.458324    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"]
Oct 31 03:29:31.492846 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.492808    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-tmp-dir\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.493073 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.492963    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-model-cache\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.493073 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.492995    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-dshm\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.493164 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.493094    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3947643c-111e-4350-9749-47643517df5c-tls-certs\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.493210 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.493170    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srnh\" (UniqueName: \"kubernetes.io/projected/3947643c-111e-4350-9749-47643517df5c-kube-api-access-8srnh\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.493210 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.493205    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-home\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.493315 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.493291    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-tmp-dir\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.493354 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.493334    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-model-cache\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.493487 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.493457    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-home\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.495381 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.495351    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-dshm\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.495627 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.495612    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3947643c-111e-4350-9749-47643517df5c-tls-certs\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.547950 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.547916    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srnh\" (UniqueName: \"kubernetes.io/projected/3947643c-111e-4350-9749-47643517df5c-kube-api-access-8srnh\") pod \"e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") " pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:31.670658 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:31.670544    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:32.050945 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:32.050911    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"]
Oct 31 03:29:32.052610 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:29:32.052582    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3947643c_111e_4350_9749_47643517df5c.slice/crio-8dbd29a8d2bc34cf115e9c38e641c77c1c6eda8fce66a645751461e3cf8d5457 WatchSource:0}: Error finding container 8dbd29a8d2bc34cf115e9c38e641c77c1c6eda8fce66a645751461e3cf8d5457: Status 404 returned error can't find the container with id 8dbd29a8d2bc34cf115e9c38e641c77c1c6eda8fce66a645751461e3cf8d5457
Oct 31 03:29:32.385123 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:32.385087    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp" event={"ID":"3947643c-111e-4350-9749-47643517df5c","Type":"ContainerStarted","Data":"e91cb74f9db26b45fdb5981cf6a4289746a7091dc9d9029b008915b25fc65e06"}
Oct 31 03:29:32.385123 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:32.385131    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp" event={"ID":"3947643c-111e-4350-9749-47643517df5c","Type":"ContainerStarted","Data":"8dbd29a8d2bc34cf115e9c38e641c77c1c6eda8fce66a645751461e3cf8d5457"}
Oct 31 03:29:32.505380 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:32.505324    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp" podStartSLOduration=1.505309191 podStartE2EDuration="1.505309191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:32.502232706 +0000 UTC m=+1314.062712459" watchObservedRunningTime="2026-10-31 03:29:32.505309191 +0000 UTC m=+1314.065788916"
Oct 31 03:29:32.670831 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:32.670749    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:41.671544 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:41.671507    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:41.684808 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:41.684774    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:42.457353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:42.457320    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:29:46.175536 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:29:46.175499    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:29:49.562354 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.562311    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"]
Oct 31 03:29:49.567594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.567568    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:49.619318 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.619278    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a16786-serving-cert\""
Oct 31 03:29:49.636194 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.636150    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-a16786-8589945596-rdp9h\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:49.636387 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.636283    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-metrics-certs\") pod \"maas-api-e2e-api-a-a16786-8589945596-rdp9h\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:49.636606 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.636583    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-maas-api-tls\") pod \"maas-api-e2e-api-a-a16786-8589945596-rdp9h\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:49.636707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.636692    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hgw\" (UniqueName: \"kubernetes.io/projected/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-kube-api-access-q6hgw\") pod \"maas-api-e2e-api-a-a16786-8589945596-rdp9h\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:49.670127 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.670086    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"]
Oct 31 03:29:49.737558 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.737520    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-maas-api-tls\") pod \"maas-api-e2e-api-a-a16786-8589945596-rdp9h\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:49.737751 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.737603    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hgw\" (UniqueName: \"kubernetes.io/projected/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-kube-api-access-q6hgw\") pod \"maas-api-e2e-api-a-a16786-8589945596-rdp9h\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:49.737751 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.737663    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-a16786-8589945596-rdp9h\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:49.737751 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:29:49.737670    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-serving-cert: secret "maas-api-e2e-api-a-a16786-serving-cert" not found
Oct 31 03:29:49.737751 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.737716    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-metrics-certs\") pod \"maas-api-e2e-api-a-a16786-8589945596-rdp9h\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:49.737751 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:29:49.737731    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-maas-api-tls podName:f1f4e579-566a-4e6d-b555-ff1ec9b121d5 nodeName:}" failed. No retries permitted until 2026-10-31 03:29:50.237714793 +0000 UTC m=+1331.798194507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-maas-api-tls") pod "maas-api-e2e-api-a-a16786-8589945596-rdp9h" (UID: "f1f4e579-566a-4e6d-b555-ff1ec9b121d5") : secret "maas-api-e2e-api-a-a16786-serving-cert" not found
Oct 31 03:29:49.738442 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.738419    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-a16786-8589945596-rdp9h\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:49.740244 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.740218    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-metrics-certs\") pod \"maas-api-e2e-api-a-a16786-8589945596-rdp9h\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:49.795198 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:49.795163    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hgw\" (UniqueName: \"kubernetes.io/projected/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-kube-api-access-q6hgw\") pod \"maas-api-e2e-api-a-a16786-8589945596-rdp9h\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:50.244860 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:50.244820    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-maas-api-tls\") pod \"maas-api-e2e-api-a-a16786-8589945596-rdp9h\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:50.247552 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:50.247524    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-maas-api-tls\") pod \"maas-api-e2e-api-a-a16786-8589945596-rdp9h\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:50.481736 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:50.481698    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:50.878210 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:50.878179    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"]
Oct 31 03:29:50.879294 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:29:50.879258    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f4e579_566a_4e6d_b555_ff1ec9b121d5.slice/crio-d3840136f67e4d7f66d2a995dbcf2f984fbc7de0e75027a9228b650f4ae2eb2a WatchSource:0}: Error finding container d3840136f67e4d7f66d2a995dbcf2f984fbc7de0e75027a9228b650f4ae2eb2a: Status 404 returned error can't find the container with id d3840136f67e4d7f66d2a995dbcf2f984fbc7de0e75027a9228b650f4ae2eb2a
Oct 31 03:29:51.501292 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:51.501255    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h" event={"ID":"f1f4e579-566a-4e6d-b555-ff1ec9b121d5","Type":"ContainerStarted","Data":"d3840136f67e4d7f66d2a995dbcf2f984fbc7de0e75027a9228b650f4ae2eb2a"}
Oct 31 03:29:52.509771 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:52.509728    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h" event={"ID":"f1f4e579-566a-4e6d-b555-ff1ec9b121d5","Type":"ContainerStarted","Data":"c6cd359e71e08974532d38d8ce61a3ba77088b3e583023769062ab111233adf4"}
Oct 31 03:29:52.510200 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:52.509898    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:29:52.623291 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:52.623228    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h" podStartSLOduration=2.706517915 podStartE2EDuration="3.62321365s" totalImagesPullingTime="916.695735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:52.620674656 +0000 UTC m=+1334.181154399" watchObservedRunningTime="2026-10-31 03:29:52.62321365 +0000 UTC m=+1334.183693381"
Oct 31 03:29:58.522052 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:29:58.522023    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:30:00.169304 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.169270    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"]
Oct 31 03:30:00.173236 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.173214    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"
Oct 31 03:30:00.194387 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.194353    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"]
Oct 31 03:30:00.198227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.198208    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"
Oct 31 03:30:00.222287 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.222258    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:30:00.268817 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.268780    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"]
Oct 31 03:30:00.280704 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.280657    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bd3ad2-4f83-43c6-9883-bbf319b88b62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr\" (UID: \"81bd3ad2-4f83-43c6-9883-bbf319b88b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"
Oct 31 03:30:00.280925 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.280826    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dchz\" (UniqueName: \"kubernetes.io/projected/81bd3ad2-4f83-43c6-9883-bbf319b88b62-kube-api-access-4dchz\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr\" (UID: \"81bd3ad2-4f83-43c6-9883-bbf319b88b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"
Oct 31 03:30:00.281125 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.281101    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f20841e-637a-4503-9f67-3faf9dcd8623-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890290-2tdsp\" (UID: \"4f20841e-637a-4503-9f67-3faf9dcd8623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"
Oct 31 03:30:00.281195 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.281137    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbd2x\" (UniqueName: \"kubernetes.io/projected/4f20841e-637a-4503-9f67-3faf9dcd8623-kube-api-access-bbd2x\") pod \"maas-api-key-cleanup-29890290-2tdsp\" (UID: \"4f20841e-637a-4503-9f67-3faf9dcd8623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"
Oct 31 03:30:00.294480 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.294441    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"]
Oct 31 03:30:00.382751 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.382655    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dchz\" (UniqueName: \"kubernetes.io/projected/81bd3ad2-4f83-43c6-9883-bbf319b88b62-kube-api-access-4dchz\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr\" (UID: \"81bd3ad2-4f83-43c6-9883-bbf319b88b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"
Oct 31 03:30:00.382950 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.382843    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f20841e-637a-4503-9f67-3faf9dcd8623-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890290-2tdsp\" (UID: \"4f20841e-637a-4503-9f67-3faf9dcd8623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"
Oct 31 03:30:00.382950 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.382871    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbd2x\" (UniqueName: \"kubernetes.io/projected/4f20841e-637a-4503-9f67-3faf9dcd8623-kube-api-access-bbd2x\") pod \"maas-api-key-cleanup-29890290-2tdsp\" (UID: \"4f20841e-637a-4503-9f67-3faf9dcd8623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"
Oct 31 03:30:00.383077 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.382963    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bd3ad2-4f83-43c6-9883-bbf319b88b62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr\" (UID: \"81bd3ad2-4f83-43c6-9883-bbf319b88b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"
Oct 31 03:30:00.383533 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.383512    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f20841e-637a-4503-9f67-3faf9dcd8623-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890290-2tdsp\" (UID: \"4f20841e-637a-4503-9f67-3faf9dcd8623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"
Oct 31 03:30:00.383702 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.383682    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bd3ad2-4f83-43c6-9883-bbf319b88b62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr\" (UID: \"81bd3ad2-4f83-43c6-9883-bbf319b88b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"
Oct 31 03:30:00.436863 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.436834    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dchz\" (UniqueName: \"kubernetes.io/projected/81bd3ad2-4f83-43c6-9883-bbf319b88b62-kube-api-access-4dchz\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr\" (UID: \"81bd3ad2-4f83-43c6-9883-bbf319b88b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"
Oct 31 03:30:00.437391 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.437368    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbd2x\" (UniqueName: \"kubernetes.io/projected/4f20841e-637a-4503-9f67-3faf9dcd8623-kube-api-access-bbd2x\") pod \"maas-api-key-cleanup-29890290-2tdsp\" (UID: \"4f20841e-637a-4503-9f67-3faf9dcd8623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"
Oct 31 03:30:00.486791 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.486757    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"
Oct 31 03:30:00.512513 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.512469    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"
Oct 31 03:30:00.862775 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.862752    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"]
Oct 31 03:30:00.864703 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:30:00.864674    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f20841e_637a_4503_9f67_3faf9dcd8623.slice/crio-0cca4b1dfa2c73ba7f9b0cf56bd5738711f5222f17c1b3236667193541a4f22d WatchSource:0}: Error finding container 0cca4b1dfa2c73ba7f9b0cf56bd5738711f5222f17c1b3236667193541a4f22d: Status 404 returned error can't find the container with id 0cca4b1dfa2c73ba7f9b0cf56bd5738711f5222f17c1b3236667193541a4f22d
Oct 31 03:30:00.911257 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:00.911224    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"]
Oct 31 03:30:00.913237 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:30:00.913198    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bd3ad2_4f83_43c6_9883_bbf319b88b62.slice/crio-9252607700d0a9833d0c05392686392ce4e019c40920c2b02c40933149059c20 WatchSource:0}: Error finding container 9252607700d0a9833d0c05392686392ce4e019c40920c2b02c40933149059c20: Status 404 returned error can't find the container with id 9252607700d0a9833d0c05392686392ce4e019c40920c2b02c40933149059c20
Oct 31 03:30:01.585542 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:01.585506    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp" event={"ID":"4f20841e-637a-4503-9f67-3faf9dcd8623","Type":"ContainerStarted","Data":"0cca4b1dfa2c73ba7f9b0cf56bd5738711f5222f17c1b3236667193541a4f22d"}
Oct 31 03:30:01.587011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:01.586984    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr" event={"ID":"81bd3ad2-4f83-43c6-9883-bbf319b88b62","Type":"ContainerStarted","Data":"9252607700d0a9833d0c05392686392ce4e019c40920c2b02c40933149059c20"}
Oct 31 03:30:06.625328 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:06.625274    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr" event={"ID":"81bd3ad2-4f83-43c6-9883-bbf319b88b62","Type":"ContainerStarted","Data":"83fe072c0f71bcf68d3925eb195718d13d31d880cab6f9220805f3dc0bf54847"}
Oct 31 03:30:06.626988 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:06.626954    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp" event={"ID":"4f20841e-637a-4503-9f67-3faf9dcd8623","Type":"ContainerStarted","Data":"6030da49031fe00651f2894d4aa04e057bd2b8ca32c098541a55ccdb13029dad"}
Oct 31 03:30:06.733788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:06.733730    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr" podStartSLOduration=1.577171267 podStartE2EDuration="6.733716758s" totalImagesPullingTime="5.156545491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:06.733016644 +0000 UTC m=+1348.293496376" watchObservedRunningTime="2026-10-31 03:30:06.733716758 +0000 UTC m=+1348.294196490"
Oct 31 03:30:06.840983 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:06.840921    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp" podStartSLOduration=1.826935283 podStartE2EDuration="6.840904985s" totalImagesPullingTime="5.013969702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:06.83994701 +0000 UTC m=+1348.400426743" watchObservedRunningTime="2026-10-31 03:30:06.840904985 +0000 UTC m=+1348.401384707"
Oct 31 03:30:07.636637 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:07.636602    5415 generic.go:356] "Generic (PLEG): container finished" podID="81bd3ad2-4f83-43c6-9883-bbf319b88b62" containerID="83fe072c0f71bcf68d3925eb195718d13d31d880cab6f9220805f3dc0bf54847" exitCode=60
Oct 31 03:30:07.637154 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:07.636650    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr" event={"ID":"81bd3ad2-4f83-43c6-9883-bbf319b88b62","Type":"ContainerDied","Data":"83fe072c0f71bcf68d3925eb195718d13d31d880cab6f9220805f3dc0bf54847"}
Oct 31 03:30:07.637154 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:07.637036    5415 scope.go:122] "RemoveContainer" containerID="83fe072c0f71bcf68d3925eb195718d13d31d880cab6f9220805f3dc0bf54847"
Oct 31 03:30:07.638676 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:07.638646    5415 generic.go:356] "Generic (PLEG): container finished" podID="4f20841e-637a-4503-9f67-3faf9dcd8623" containerID="6030da49031fe00651f2894d4aa04e057bd2b8ca32c098541a55ccdb13029dad" exitCode=60
Oct 31 03:30:07.638815 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:07.638677    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp" event={"ID":"4f20841e-637a-4503-9f67-3faf9dcd8623","Type":"ContainerDied","Data":"6030da49031fe00651f2894d4aa04e057bd2b8ca32c098541a55ccdb13029dad"}
Oct 31 03:30:07.639234 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:07.639217    5415 scope.go:122] "RemoveContainer" containerID="6030da49031fe00651f2894d4aa04e057bd2b8ca32c098541a55ccdb13029dad"
Oct 31 03:30:08.647958 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:08.647913    5415 generic.go:356] "Generic (PLEG): container finished" podID="81bd3ad2-4f83-43c6-9883-bbf319b88b62" containerID="70cd6e3e8d48355bb99d3c66156ff0e8b5e142e40afd85320297b3ea0bfe89eb" exitCode=60
Oct 31 03:30:08.648580 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:08.647980    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr" event={"ID":"81bd3ad2-4f83-43c6-9883-bbf319b88b62","Type":"ContainerDied","Data":"70cd6e3e8d48355bb99d3c66156ff0e8b5e142e40afd85320297b3ea0bfe89eb"}
Oct 31 03:30:08.648580 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:08.648044    5415 scope.go:122] "RemoveContainer" containerID="83fe072c0f71bcf68d3925eb195718d13d31d880cab6f9220805f3dc0bf54847"
Oct 31 03:30:08.648580 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:08.648319    5415 scope.go:122] "RemoveContainer" containerID="70cd6e3e8d48355bb99d3c66156ff0e8b5e142e40afd85320297b3ea0bfe89eb"
Oct 31 03:30:08.648739 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:30:08.648648    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr_odh-ai-gateway-infra(81bd3ad2-4f83-43c6-9883-bbf319b88b62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr" podUID="81bd3ad2-4f83-43c6-9883-bbf319b88b62"
Oct 31 03:30:08.650340 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:08.650315    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp" event={"ID":"4f20841e-637a-4503-9f67-3faf9dcd8623","Type":"ContainerStarted","Data":"09ed9d68dadd39c3ad61bbd4c438b1dba33d9a2ab77f5a639a99875824f044f0"}
Oct 31 03:30:09.663590 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:09.663547    5415 generic.go:356] "Generic (PLEG): container finished" podID="4f20841e-637a-4503-9f67-3faf9dcd8623" containerID="09ed9d68dadd39c3ad61bbd4c438b1dba33d9a2ab77f5a639a99875824f044f0" exitCode=60
Oct 31 03:30:09.664227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:09.663720    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp" event={"ID":"4f20841e-637a-4503-9f67-3faf9dcd8623","Type":"ContainerDied","Data":"09ed9d68dadd39c3ad61bbd4c438b1dba33d9a2ab77f5a639a99875824f044f0"}
Oct 31 03:30:09.664227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:09.663865    5415 scope.go:122] "RemoveContainer" containerID="6030da49031fe00651f2894d4aa04e057bd2b8ca32c098541a55ccdb13029dad"
Oct 31 03:30:09.665233 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:09.665206    5415 scope.go:122] "RemoveContainer" containerID="09ed9d68dadd39c3ad61bbd4c438b1dba33d9a2ab77f5a639a99875824f044f0"
Oct 31 03:30:09.665746 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:30:09.665705    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890290-2tdsp_odh-ai-gateway-infra(4f20841e-637a-4503-9f67-3faf9dcd8623)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp" podUID="4f20841e-637a-4503-9f67-3faf9dcd8623"
Oct 31 03:30:09.668668 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:09.668642    5415 scope.go:122] "RemoveContainer" containerID="70cd6e3e8d48355bb99d3c66156ff0e8b5e142e40afd85320297b3ea0bfe89eb"
Oct 31 03:30:09.669350 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:30:09.669320    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr_odh-ai-gateway-infra(81bd3ad2-4f83-43c6-9883-bbf319b88b62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr" podUID="81bd3ad2-4f83-43c6-9883-bbf319b88b62"
Oct 31 03:30:10.716972 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:10.716935    5415 scope.go:122] "RemoveContainer" containerID="09ed9d68dadd39c3ad61bbd4c438b1dba33d9a2ab77f5a639a99875824f044f0"
Oct 31 03:30:10.718203 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:30:10.718169    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890290-2tdsp_odh-ai-gateway-infra(4f20841e-637a-4503-9f67-3faf9dcd8623)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp" podUID="4f20841e-637a-4503-9f67-3faf9dcd8623"
Oct 31 03:30:14.204623 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.204587    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"]
Oct 31 03:30:14.210372 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.210349    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.297939 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.297894    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-model-cache\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.298152 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.297949    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tmp-dir\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.298227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.298179    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-home\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.298227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.298213    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tls-certs\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.298307 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.298246    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh9ph\" (UniqueName: \"kubernetes.io/projected/37142a22-1f4a-4bbe-ac5b-72a3ff276832-kube-api-access-dh9ph\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.298445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.298417    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-dshm\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.307256 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.307223    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd5b85\"/\"distinct-2-w3-dd5b85-kserve-self-signed-certs\""
Oct 31 03:30:14.310231 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.310199    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"]
Oct 31 03:30:14.400165 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.400132    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-dshm\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.400385 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.400297    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-model-cache\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.400385 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.400344    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tmp-dir\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.400544 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.400514    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-home\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.400635 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.400541    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tls-certs\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.400635 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.400571    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9ph\" (UniqueName: \"kubernetes.io/projected/37142a22-1f4a-4bbe-ac5b-72a3ff276832-kube-api-access-dh9ph\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.401011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.400987    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tmp-dir\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.401280 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.401258    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-model-cache\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.401358 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.401293    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-home\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.405198 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.405169    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-dshm\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.406791 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.406763    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tls-certs\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.457296 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.457213    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh9ph\" (UniqueName: \"kubernetes.io/projected/37142a22-1f4a-4bbe-ac5b-72a3ff276832-kube-api-access-dh9ph\") pod \"distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") " pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.524155 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.524112    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:14.918657 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:14.918625    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"]
Oct 31 03:30:14.920564 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:30:14.920535    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37142a22_1f4a_4bbe_ac5b_72a3ff276832.slice/crio-9353fec719c65c7547cf54664a9e38392807f58af6fcccd6a43664b513e5f003 WatchSource:0}: Error finding container 9353fec719c65c7547cf54664a9e38392807f58af6fcccd6a43664b513e5f003: Status 404 returned error can't find the container with id 9353fec719c65c7547cf54664a9e38392807f58af6fcccd6a43664b513e5f003
Oct 31 03:30:15.755442 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:15.755402    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq" event={"ID":"37142a22-1f4a-4bbe-ac5b-72a3ff276832","Type":"ContainerStarted","Data":"04f60ce4e7af25a76872ae3f7445c36f1b52bfb642352ddae9ac2aece396db1f"}
Oct 31 03:30:15.755865 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:15.755452    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq" event={"ID":"37142a22-1f4a-4bbe-ac5b-72a3ff276832","Type":"ContainerStarted","Data":"9353fec719c65c7547cf54664a9e38392807f58af6fcccd6a43664b513e5f003"}
Oct 31 03:30:15.865962 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:15.865905    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq" podStartSLOduration=1.865863322 podStartE2EDuration="1.865863322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:15.864986888 +0000 UTC m=+1357.425466647" watchObservedRunningTime="2026-10-31 03:30:15.865863322 +0000 UTC m=+1357.426343053"
Oct 31 03:30:16.177504 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:30:16.177453    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:30:16.524860 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:16.524776    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:21.599112 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:21.599080    5415 scope.go:122] "RemoveContainer" containerID="09ed9d68dadd39c3ad61bbd4c438b1dba33d9a2ab77f5a639a99875824f044f0"
Oct 31 03:30:22.606216 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:22.606189    5415 scope.go:122] "RemoveContainer" containerID="70cd6e3e8d48355bb99d3c66156ff0e8b5e142e40afd85320297b3ea0bfe89eb"
Oct 31 03:30:22.800076 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:22.800040    5415 generic.go:356] "Generic (PLEG): container finished" podID="4f20841e-637a-4503-9f67-3faf9dcd8623" containerID="e577bc00f34d132d1ed3eb20cd55d391372cdc6f9fdf8fe561e37094bd7f0ac9" exitCode=60
Oct 31 03:30:22.800266 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:22.800117    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp" event={"ID":"4f20841e-637a-4503-9f67-3faf9dcd8623","Type":"ContainerDied","Data":"e577bc00f34d132d1ed3eb20cd55d391372cdc6f9fdf8fe561e37094bd7f0ac9"}
Oct 31 03:30:22.800266 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:22.800182    5415 scope.go:122] "RemoveContainer" containerID="09ed9d68dadd39c3ad61bbd4c438b1dba33d9a2ab77f5a639a99875824f044f0"
Oct 31 03:30:22.800583 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:22.800565    5415 scope.go:122] "RemoveContainer" containerID="e577bc00f34d132d1ed3eb20cd55d391372cdc6f9fdf8fe561e37094bd7f0ac9"
Oct 31 03:30:22.800949 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:30:22.800930    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890290-2tdsp_odh-ai-gateway-infra(4f20841e-637a-4503-9f67-3faf9dcd8623)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp" podUID="4f20841e-637a-4503-9f67-3faf9dcd8623"
Oct 31 03:30:23.811359 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:23.811259    5415 generic.go:356] "Generic (PLEG): container finished" podID="81bd3ad2-4f83-43c6-9883-bbf319b88b62" containerID="c62f2fb2a4fe4bc3bd78d371e9396fed2809b3ef9ce6e54adf036a8a570f9adc" exitCode=60
Oct 31 03:30:23.811359 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:23.811324    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr" event={"ID":"81bd3ad2-4f83-43c6-9883-bbf319b88b62","Type":"ContainerDied","Data":"c62f2fb2a4fe4bc3bd78d371e9396fed2809b3ef9ce6e54adf036a8a570f9adc"}
Oct 31 03:30:23.811816 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:23.811381    5415 scope.go:122] "RemoveContainer" containerID="70cd6e3e8d48355bb99d3c66156ff0e8b5e142e40afd85320297b3ea0bfe89eb"
Oct 31 03:30:23.811816 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:23.811793    5415 scope.go:122] "RemoveContainer" containerID="c62f2fb2a4fe4bc3bd78d371e9396fed2809b3ef9ce6e54adf036a8a570f9adc"
Oct 31 03:30:23.812187 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:30:23.812160    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr_odh-ai-gateway-infra(81bd3ad2-4f83-43c6-9883-bbf319b88b62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr" podUID="81bd3ad2-4f83-43c6-9883-bbf319b88b62"
Oct 31 03:30:23.919841 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:23.919797    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"]
Oct 31 03:30:24.090369 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.090340    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"
Oct 31 03:30:24.252927 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.252891    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f20841e-637a-4503-9f67-3faf9dcd8623-kube-api-access-bbd2x\" (UniqueName: \"kubernetes.io/projected/4f20841e-637a-4503-9f67-3faf9dcd8623-kube-api-access-bbd2x\") pod \"4f20841e-637a-4503-9f67-3faf9dcd8623\" (UID: \"4f20841e-637a-4503-9f67-3faf9dcd8623\") "
Oct 31 03:30:24.253099 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.253051    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f20841e-637a-4503-9f67-3faf9dcd8623-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f20841e-637a-4503-9f67-3faf9dcd8623-maas-api-service-ca\") pod \"4f20841e-637a-4503-9f67-3faf9dcd8623\" (UID: \"4f20841e-637a-4503-9f67-3faf9dcd8623\") "
Oct 31 03:30:24.253401 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.253375    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f20841e-637a-4503-9f67-3faf9dcd8623-maas-api-service-ca" pod "4f20841e-637a-4503-9f67-3faf9dcd8623" (UID: "4f20841e-637a-4503-9f67-3faf9dcd8623"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:24.253647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.253630    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f20841e-637a-4503-9f67-3faf9dcd8623-maas-api-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:24.255090 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.255064    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f20841e-637a-4503-9f67-3faf9dcd8623-kube-api-access-bbd2x" pod "4f20841e-637a-4503-9f67-3faf9dcd8623" (UID: "4f20841e-637a-4503-9f67-3faf9dcd8623"). InnerVolumeSpecName "kube-api-access-bbd2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:24.355054 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.355020    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbd2x\" (UniqueName: \"kubernetes.io/projected/4f20841e-637a-4503-9f67-3faf9dcd8623-kube-api-access-bbd2x\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:24.524466 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.524432    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:24.538447 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.538417    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:24.819154 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.819114    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp" event={"ID":"4f20841e-637a-4503-9f67-3faf9dcd8623","Type":"ContainerDied","Data":"0cca4b1dfa2c73ba7f9b0cf56bd5738711f5222f17c1b3236667193541a4f22d"}
Oct 31 03:30:24.819609 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.819183    5415 scope.go:122] "RemoveContainer" containerID="e577bc00f34d132d1ed3eb20cd55d391372cdc6f9fdf8fe561e37094bd7f0ac9"
Oct 31 03:30:24.819609 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.819133    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"
Oct 31 03:30:24.835078 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.835054    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 03:30:24.931838 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:24.931754    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"]
Oct 31 03:30:25.086865 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:25.086804    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"]
Oct 31 03:30:25.090914 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:25.090784    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-2tdsp"]
Oct 31 03:30:25.125855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:25.125825    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"
Oct 31 03:30:25.266989 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:25.266899    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81bd3ad2-4f83-43c6-9883-bbf319b88b62-kube-api-access-4dchz\" (UniqueName: \"kubernetes.io/projected/81bd3ad2-4f83-43c6-9883-bbf319b88b62-kube-api-access-4dchz\") pod \"81bd3ad2-4f83-43c6-9883-bbf319b88b62\" (UID: \"81bd3ad2-4f83-43c6-9883-bbf319b88b62\") "
Oct 31 03:30:25.267172 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:25.267147    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81bd3ad2-4f83-43c6-9883-bbf319b88b62-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bd3ad2-4f83-43c6-9883-bbf319b88b62-maas-api-service-ca\") pod \"81bd3ad2-4f83-43c6-9883-bbf319b88b62\" (UID: \"81bd3ad2-4f83-43c6-9883-bbf319b88b62\") "
Oct 31 03:30:25.267472 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:25.267450    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81bd3ad2-4f83-43c6-9883-bbf319b88b62-maas-api-service-ca" pod "81bd3ad2-4f83-43c6-9883-bbf319b88b62" (UID: "81bd3ad2-4f83-43c6-9883-bbf319b88b62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:25.267676 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:25.267657    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bd3ad2-4f83-43c6-9883-bbf319b88b62-maas-api-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:25.269041 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:25.269021    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81bd3ad2-4f83-43c6-9883-bbf319b88b62-kube-api-access-4dchz" pod "81bd3ad2-4f83-43c6-9883-bbf319b88b62" (UID: "81bd3ad2-4f83-43c6-9883-bbf319b88b62"). InnerVolumeSpecName "kube-api-access-4dchz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:25.368821 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:25.368787    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dchz\" (UniqueName: \"kubernetes.io/projected/81bd3ad2-4f83-43c6-9883-bbf319b88b62-kube-api-access-4dchz\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:25.830253 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:25.830222    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"
Oct 31 03:30:25.830253 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:25.830246    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr" event={"ID":"81bd3ad2-4f83-43c6-9883-bbf319b88b62","Type":"ContainerDied","Data":"9252607700d0a9833d0c05392686392ce4e019c40920c2b02c40933149059c20"}
Oct 31 03:30:25.830822 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:25.830319    5415 scope.go:122] "RemoveContainer" containerID="c62f2fb2a4fe4bc3bd78d371e9396fed2809b3ef9ce6e54adf036a8a570f9adc"
Oct 31 03:30:25.985148 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:25.985112    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"]
Oct 31 03:30:26.032172 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:26.032136    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890290-vk6qr"]
Oct 31 03:30:26.607117 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:26.607083    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f20841e-637a-4503-9f67-3faf9dcd8623" path="/var/lib/kubelet/pods/4f20841e-637a-4503-9f67-3faf9dcd8623/volumes"
Oct 31 03:30:26.607608 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:26.607594    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81bd3ad2-4f83-43c6-9883-bbf319b88b62" path="/var/lib/kubelet/pods/81bd3ad2-4f83-43c6-9883-bbf319b88b62/volumes"
Oct 31 03:30:37.183985 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.183949    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"]
Oct 31 03:30:37.184566 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184548    5415 cpu_manager.go:414] "containerMap: removing container" podUID="4f20841e-637a-4503-9f67-3faf9dcd8623" containerName="cleanup"
Oct 31 03:30:37.184647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184568    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f20841e-637a-4503-9f67-3faf9dcd8623" containerName="cleanup"
Oct 31 03:30:37.184647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184582    5415 cpu_manager.go:414] "containerMap: removing container" podUID="81bd3ad2-4f83-43c6-9883-bbf319b88b62" containerName="cleanup"
Oct 31 03:30:37.184647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184590    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81bd3ad2-4f83-43c6-9883-bbf319b88b62" containerName="cleanup"
Oct 31 03:30:37.184647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184609    5415 cpu_manager.go:414] "containerMap: removing container" podUID="81bd3ad2-4f83-43c6-9883-bbf319b88b62" containerName="cleanup"
Oct 31 03:30:37.184647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184618    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81bd3ad2-4f83-43c6-9883-bbf319b88b62" containerName="cleanup"
Oct 31 03:30:37.184647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184630    5415 cpu_manager.go:414] "containerMap: removing container" podUID="81bd3ad2-4f83-43c6-9883-bbf319b88b62" containerName="cleanup"
Oct 31 03:30:37.184647 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184635    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81bd3ad2-4f83-43c6-9883-bbf319b88b62" containerName="cleanup"
Oct 31 03:30:37.184992 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184661    5415 cpu_manager.go:414] "containerMap: removing container" podUID="4f20841e-637a-4503-9f67-3faf9dcd8623" containerName="cleanup"
Oct 31 03:30:37.184992 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184667    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f20841e-637a-4503-9f67-3faf9dcd8623" containerName="cleanup"
Oct 31 03:30:37.184992 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184796    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f20841e-637a-4503-9f67-3faf9dcd8623" containerName="cleanup"
Oct 31 03:30:37.184992 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184813    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f20841e-637a-4503-9f67-3faf9dcd8623" containerName="cleanup"
Oct 31 03:30:37.184992 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184824    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="81bd3ad2-4f83-43c6-9883-bbf319b88b62" containerName="cleanup"
Oct 31 03:30:37.184992 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184833    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f20841e-637a-4503-9f67-3faf9dcd8623" containerName="cleanup"
Oct 31 03:30:37.184992 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184843    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="81bd3ad2-4f83-43c6-9883-bbf319b88b62" containerName="cleanup"
Oct 31 03:30:37.184992 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.184850    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="81bd3ad2-4f83-43c6-9883-bbf319b88b62" containerName="cleanup"
Oct 31 03:30:37.193102 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.193073    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.231411 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.231378    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"]
Oct 31 03:30:37.249511 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.249482    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d4fdeb-serving-cert\""
Oct 31 03:30:37.327621 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.327583    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-maas-api-tls\") pod \"maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.327769 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.327640    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528321a5-da5c-415b-b3f2-4030a4945aa2-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.327827 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.327801    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pgzj\" (UniqueName: \"kubernetes.io/projected/528321a5-da5c-415b-b3f2-4030a4945aa2-kube-api-access-4pgzj\") pod \"maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.328119 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.328098    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-metrics-certs\") pod \"maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.432467 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.432425    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-metrics-certs\") pod \"maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.433319 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.433292    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-maas-api-tls\") pod \"maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.433907 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.433854    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528321a5-da5c-415b-b3f2-4030a4945aa2-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.434100 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.433952    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgzj\" (UniqueName: \"kubernetes.io/projected/528321a5-da5c-415b-b3f2-4030a4945aa2-kube-api-access-4pgzj\") pod \"maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.436675 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.434887    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528321a5-da5c-415b-b3f2-4030a4945aa2-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.436857 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.436134    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-metrics-certs\") pod \"maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.451900 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.449268    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-maas-api-tls\") pod \"maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.489012 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.488982    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgzj\" (UniqueName: \"kubernetes.io/projected/528321a5-da5c-415b-b3f2-4030a4945aa2-kube-api-access-4pgzj\") pod \"maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.507183 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.507152    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:37.904906 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:37.902374    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"]
Oct 31 03:30:38.918686 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:38.918650    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8" event={"ID":"528321a5-da5c-415b-b3f2-4030a4945aa2","Type":"ContainerStarted","Data":"cb7d42a59b615787ae1eed13d226a79b472121f2967b3b951aa7987a5089b539"}
Oct 31 03:30:38.918686 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:38.918706    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8" event={"ID":"528321a5-da5c-415b-b3f2-4030a4945aa2","Type":"ContainerStarted","Data":"5f99886a6dee55043eba3eb9fcd49883f11245a7d6e7171d3aa1082122a89224"}
Oct 31 03:30:38.919226 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:38.918783    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:39.042562 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:39.042484    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8" podStartSLOduration=1.139415881 podStartE2EDuration="2.04246457s" totalImagesPullingTime="903.048689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:39.040591715 +0000 UTC m=+1380.601071446" watchObservedRunningTime="2026-10-31 03:30:39.04246457 +0000 UTC m=+1380.602944300"
Oct 31 03:30:44.931611 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:30:44.931577    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:30:46.176051 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:30:46.176012    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:31:02.445463 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:02.445425    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7199-4079d1"]
Oct 31 03:31:02.446100 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:02.446083    5415 cpu_manager.go:414] "containerMap: removing container" podUID="4f20841e-637a-4503-9f67-3faf9dcd8623" containerName="cleanup"
Oct 31 03:31:02.446174 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:02.446102    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f20841e-637a-4503-9f67-3faf9dcd8623" containerName="cleanup"
Oct 31 03:31:02.449840 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:02.449817    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7199-4079d1"
Oct 31 03:31:02.504121 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:02.504090    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7199-4079d1"]
Oct 31 03:31:02.543511 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:02.543475    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c4jjr\""
Oct 31 03:31:02.576639 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:02.576596    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l286b\" (UniqueName: \"kubernetes.io/projected/b6b5732f-a025-413f-82a5-741a692149cd-kube-api-access-l286b\") pod \"test-curl-7199-4079d1\" (UID: \"b6b5732f-a025-413f-82a5-741a692149cd\") " pod="opendatahub/test-curl-7199-4079d1"
Oct 31 03:31:02.678509 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:02.678455    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l286b\" (UniqueName: \"kubernetes.io/projected/b6b5732f-a025-413f-82a5-741a692149cd-kube-api-access-l286b\") pod \"test-curl-7199-4079d1\" (UID: \"b6b5732f-a025-413f-82a5-741a692149cd\") " pod="opendatahub/test-curl-7199-4079d1"
Oct 31 03:31:02.739973 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:02.739862    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l286b\" (UniqueName: \"kubernetes.io/projected/b6b5732f-a025-413f-82a5-741a692149cd-kube-api-access-l286b\") pod \"test-curl-7199-4079d1\" (UID: \"b6b5732f-a025-413f-82a5-741a692149cd\") " pod="opendatahub/test-curl-7199-4079d1"
Oct 31 03:31:02.762858 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:02.762823    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7199-4079d1"
Oct 31 03:31:03.136465 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:03.136440    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7199-4079d1"]
Oct 31 03:31:03.138726 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:31:03.138693    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6b5732f_a025_413f_82a5_741a692149cd.slice/crio-b48a18b76c2d570b38a361f9aa58b09ea1f781909ef6f1cc0360c25c9c8ed3b7 WatchSource:0}: Error finding container b48a18b76c2d570b38a361f9aa58b09ea1f781909ef6f1cc0360c25c9c8ed3b7: Status 404 returned error can't find the container with id b48a18b76c2d570b38a361f9aa58b09ea1f781909ef6f1cc0360c25c9c8ed3b7
Oct 31 03:31:04.074188 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:04.074151    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7199-4079d1" event={"ID":"b6b5732f-a025-413f-82a5-741a692149cd","Type":"ContainerStarted","Data":"b48a18b76c2d570b38a361f9aa58b09ea1f781909ef6f1cc0360c25c9c8ed3b7"}
Oct 31 03:31:05.081793 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:05.081745    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7199-4079d1" event={"ID":"b6b5732f-a025-413f-82a5-741a692149cd","Type":"ContainerStarted","Data":"7db941164dd4a704a26d87bf7d3e9e0924bbbecb9aecdf5eedf1be3794261ec2"}
Oct 31 03:31:05.187513 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:05.187453    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7199-4079d1" podStartSLOduration=1.925307628 podStartE2EDuration="3.187436777s" totalImagesPullingTime="1.262129149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:05.186868608 +0000 UTC m=+1406.747348339" watchObservedRunningTime="2026-10-31 03:31:05.187436777 +0000 UTC m=+1406.747916512"
Oct 31 03:31:06.684687 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:06.684651    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7199-4079d1"]
Oct 31 03:31:06.690866 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:06.690831    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7199-4079d1"]
Oct 31 03:31:07.094961 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:07.094903    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7199-4079d1" podUID="b6b5732f-a025-413f-82a5-741a692149cd" containerName="test-curl-7199-4079d1" containerID="cri-o://7db941164dd4a704a26d87bf7d3e9e0924bbbecb9aecdf5eedf1be3794261ec2" gracePeriod=2
Oct 31 03:31:07.143396 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:31:07.143349    5415 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7199-4079d1\" is forbidden: User \"system:node:ip-10-0-139-164.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-164.us-west-2.compute.internal' and this object" podUID="b6b5732f-a025-413f-82a5-741a692149cd" pod="opendatahub/test-curl-7199-4079d1"
Oct 31 03:31:08.650545 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:31:08.650505    5415 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7199-4079d1\" is forbidden: User \"system:node:ip-10-0-139-164.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-164.us-west-2.compute.internal' and this object" podUID="b6b5732f-a025-413f-82a5-741a692149cd" pod="opendatahub/test-curl-7199-4079d1"
Oct 31 03:31:09.358579 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:09.358511    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7199-4079d1"
Oct 31 03:31:09.370141 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:09.370111    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b5732f-a025-413f-82a5-741a692149cd-kube-api-access-l286b\" (UniqueName: \"kubernetes.io/projected/b6b5732f-a025-413f-82a5-741a692149cd-kube-api-access-l286b\") pod \"b6b5732f-a025-413f-82a5-741a692149cd\" (UID: \"b6b5732f-a025-413f-82a5-741a692149cd\") "
Oct 31 03:31:09.372241 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:09.372212    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b5732f-a025-413f-82a5-741a692149cd-kube-api-access-l286b" pod "b6b5732f-a025-413f-82a5-741a692149cd" (UID: "b6b5732f-a025-413f-82a5-741a692149cd"). InnerVolumeSpecName "kube-api-access-l286b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:09.471663 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:09.471620    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l286b\" (UniqueName: \"kubernetes.io/projected/b6b5732f-a025-413f-82a5-741a692149cd-kube-api-access-l286b\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:10.113677 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.113639    5415 generic.go:356] "Generic (PLEG): container finished" podID="b6b5732f-a025-413f-82a5-741a692149cd" containerID="7db941164dd4a704a26d87bf7d3e9e0924bbbecb9aecdf5eedf1be3794261ec2" exitCode=137
Oct 31 03:31:10.114168 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.113684    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7199-4079d1"
Oct 31 03:31:10.114168 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.113737    5415 scope.go:122] "RemoveContainer" containerID="7db941164dd4a704a26d87bf7d3e9e0924bbbecb9aecdf5eedf1be3794261ec2"
Oct 31 03:31:10.123166 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.123144    5415 scope.go:122] "RemoveContainer" containerID="7db941164dd4a704a26d87bf7d3e9e0924bbbecb9aecdf5eedf1be3794261ec2"
Oct 31 03:31:10.123449 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:31:10.123432    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db941164dd4a704a26d87bf7d3e9e0924bbbecb9aecdf5eedf1be3794261ec2\": container with ID starting with 7db941164dd4a704a26d87bf7d3e9e0924bbbecb9aecdf5eedf1be3794261ec2 not found: ID does not exist" containerID="7db941164dd4a704a26d87bf7d3e9e0924bbbecb9aecdf5eedf1be3794261ec2"
Oct 31 03:31:10.123494 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.123459    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db941164dd4a704a26d87bf7d3e9e0924bbbecb9aecdf5eedf1be3794261ec2"} err="failed to get container status \"7db941164dd4a704a26d87bf7d3e9e0924bbbecb9aecdf5eedf1be3794261ec2\": rpc error: code = NotFound desc = could not find container \"7db941164dd4a704a26d87bf7d3e9e0924bbbecb9aecdf5eedf1be3794261ec2\": container with ID starting with 7db941164dd4a704a26d87bf7d3e9e0924bbbecb9aecdf5eedf1be3794261ec2 not found: ID does not exist"
Oct 31 03:31:10.170573 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:31:10.170527    5415 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7199-4079d1\" is forbidden: User \"system:node:ip-10-0-139-164.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-164.us-west-2.compute.internal' and this object" podUID="b6b5732f-a025-413f-82a5-741a692149cd" pod="opendatahub/test-curl-7199-4079d1"
Oct 31 03:31:10.606721 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.606684    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b5732f-a025-413f-82a5-741a692149cd" path="/var/lib/kubelet/pods/b6b5732f-a025-413f-82a5-741a692149cd/volumes"
Oct 31 03:31:10.607093 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.607076    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"]
Oct 31 03:31:10.607868 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.607848    5415 cpu_manager.go:414] "containerMap: removing container" podUID="b6b5732f-a025-413f-82a5-741a692149cd" containerName="test-curl-7199-4079d1"
Oct 31 03:31:10.607926 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.607872    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b5732f-a025-413f-82a5-741a692149cd" containerName="test-curl-7199-4079d1"
Oct 31 03:31:10.608028 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.608017    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b5732f-a025-413f-82a5-741a692149cd" containerName="test-curl-7199-4079d1"
Oct 31 03:31:10.612995 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.612976    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.662438 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.662402    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-dd5b85-openshift-default-dockercfg-b24df\""
Oct 31 03:31:10.692729 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.692690    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.692729 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.692724    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.692972 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.692895    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.692972 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.692955    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.693113 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.693063    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017e3f11-748c-46c0-b54d-33112be20591-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.693169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.693126    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.693261 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.693242    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017e3f11-748c-46c0-b54d-33112be20591-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.693329 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.693277    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.693429 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.693408    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88hhs\" (UniqueName: \"kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-kube-api-access-88hhs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.718483 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.718448    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"]
Oct 31 03:31:10.794592 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.794552    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017e3f11-748c-46c0-b54d-33112be20591-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.794793 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.794696    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.794793 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.794728    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017e3f11-748c-46c0-b54d-33112be20591-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.794793 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.794748    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.795020 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.794810    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88hhs\" (UniqueName: \"kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-kube-api-access-88hhs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.795020 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.794868    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.795020 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.794927    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.795020 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.794960    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.795020 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.794998    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.795274 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.795130    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.795342 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.795323    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.795443 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.795423    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.795500 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.795470    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017e3f11-748c-46c0-b54d-33112be20591-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.795558 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.795470    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.797220 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.797198    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.797367 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.797349    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017e3f11-748c-46c0-b54d-33112be20591-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.849364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.849327    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.849521 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.849382    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88hhs\" (UniqueName: \"kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-kube-api-access-88hhs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:10.927322 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:10.927235    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:11.348742 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:11.348705    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"]
Oct 31 03:31:11.350307 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:31:11.350280    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017e3f11_748c_46c0_b54d_33112be20591.slice/crio-282fd76f76265f2084b6bc66376fd8cca76c131633315fb0b3b613715e4ecf5d WatchSource:0}: Error finding container 282fd76f76265f2084b6bc66376fd8cca76c131633315fb0b3b613715e4ecf5d: Status 404 returned error can't find the container with id 282fd76f76265f2084b6bc66376fd8cca76c131633315fb0b3b613715e4ecf5d
Oct 31 03:31:11.352829 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:11.352797    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:31:11.352915 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:11.352868    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:31:11.352951 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:11.352914    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:31:12.128101 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:12.128065    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg" event={"ID":"017e3f11-748c-46c0-b54d-33112be20591","Type":"ContainerStarted","Data":"d182ca4ffd5f6689fd7eab051ff09863ca36bfa395816af241599dee3970a5d9"}
Oct 31 03:31:12.128290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:12.128118    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg" event={"ID":"017e3f11-748c-46c0-b54d-33112be20591","Type":"ContainerStarted","Data":"282fd76f76265f2084b6bc66376fd8cca76c131633315fb0b3b613715e4ecf5d"}
Oct 31 03:31:12.336411 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:12.336339    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg" podStartSLOduration=2.336318414 podStartE2EDuration="2.336318414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:12.287339856 +0000 UTC m=+1413.847819602" watchObservedRunningTime="2026-10-31 03:31:12.336318414 +0000 UTC m=+1413.896798144"
Oct 31 03:31:12.928297 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:12.928256    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:12.930165 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:12.930141    5415 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 03:31:12.930281 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:12.930189    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg" podUID="017e3f11-748c-46c0-b54d-33112be20591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 03:31:13.928227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:13.928187    5415 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 03:31:13.928387 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:13.928248    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg" podUID="017e3f11-748c-46c0-b54d-33112be20591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 03:31:14.927756 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:14.927717    5415 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 03:31:14.927972 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:14.927772    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg" podUID="017e3f11-748c-46c0-b54d-33112be20591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 03:31:15.928171 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:15.928135    5415 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 03:31:15.928586 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:15.928203    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg" podUID="017e3f11-748c-46c0-b54d-33112be20591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 03:31:16.201433 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:31:16.201339    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:31:16.928030 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:16.927992    5415 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 03:31:16.928523 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:16.928058    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg" podUID="017e3f11-748c-46c0-b54d-33112be20591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 03:31:17.928547 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:17.928504    5415 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 03:31:17.929093 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:17.928572    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg" podUID="017e3f11-748c-46c0-b54d-33112be20591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 03:31:18.933274 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:18.933242    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:18.933813 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:18.933612    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:18.934532 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:18.934508    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:31:34.840562 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:34.840512    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"]
Oct 31 03:31:34.844916 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:34.844892    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:34.894499 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:34.894464    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-070d07-serving-cert\""
Oct 31 03:31:34.945723 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:34.945680    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"]
Oct 31 03:31:34.969630 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:34.969594    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-metrics-certs\") pod \"maas-api-e2e-api-b-070d07-78bc48c648-wlj98\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:34.969809 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:34.969667    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc77befa-5d3a-421f-8076-25786aac3569-openshift-service-ca\") pod \"maas-api-e2e-api-b-070d07-78bc48c648-wlj98\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:34.969809 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:34.969762    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-maas-api-tls\") pod \"maas-api-e2e-api-b-070d07-78bc48c648-wlj98\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:34.969933 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:34.969915    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gg9n\" (UniqueName: \"kubernetes.io/projected/dc77befa-5d3a-421f-8076-25786aac3569-kube-api-access-2gg9n\") pod \"maas-api-e2e-api-b-070d07-78bc48c648-wlj98\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:35.071296 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:35.071260    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gg9n\" (UniqueName: \"kubernetes.io/projected/dc77befa-5d3a-421f-8076-25786aac3569-kube-api-access-2gg9n\") pod \"maas-api-e2e-api-b-070d07-78bc48c648-wlj98\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:35.071471 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:35.071370    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-metrics-certs\") pod \"maas-api-e2e-api-b-070d07-78bc48c648-wlj98\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:35.071471 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:35.071428    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc77befa-5d3a-421f-8076-25786aac3569-openshift-service-ca\") pod \"maas-api-e2e-api-b-070d07-78bc48c648-wlj98\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:35.071471 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:35.071462    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-maas-api-tls\") pod \"maas-api-e2e-api-b-070d07-78bc48c648-wlj98\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:35.071578 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:31:35.071569    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-serving-cert: secret "maas-api-e2e-api-b-070d07-serving-cert" not found
Oct 31 03:31:35.071635 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:31:35.071625    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-maas-api-tls podName:dc77befa-5d3a-421f-8076-25786aac3569 nodeName:}" failed. No retries permitted until 2026-10-31 03:31:35.571607589 +0000 UTC m=+1437.132087298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-maas-api-tls") pod "maas-api-e2e-api-b-070d07-78bc48c648-wlj98" (UID: "dc77befa-5d3a-421f-8076-25786aac3569") : secret "maas-api-e2e-api-b-070d07-serving-cert" not found
Oct 31 03:31:35.072146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:35.072123    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc77befa-5d3a-421f-8076-25786aac3569-openshift-service-ca\") pod \"maas-api-e2e-api-b-070d07-78bc48c648-wlj98\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:35.073997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:35.073972    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-metrics-certs\") pod \"maas-api-e2e-api-b-070d07-78bc48c648-wlj98\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:35.128869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:35.128767    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gg9n\" (UniqueName: \"kubernetes.io/projected/dc77befa-5d3a-421f-8076-25786aac3569-kube-api-access-2gg9n\") pod \"maas-api-e2e-api-b-070d07-78bc48c648-wlj98\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:35.579397 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:35.579300    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-maas-api-tls\") pod \"maas-api-e2e-api-b-070d07-78bc48c648-wlj98\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:35.581787 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:35.581765    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-maas-api-tls\") pod \"maas-api-e2e-api-b-070d07-78bc48c648-wlj98\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:35.760294 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:35.760158    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:36.130277 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:36.130248    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"]
Oct 31 03:31:36.132321 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:31:36.132295    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc77befa_5d3a_421f_8076_25786aac3569.slice/crio-5f971e976417078f31f8a447d04d6c8bfd2dfea7ff15b22ea37ce5c330931462 WatchSource:0}: Error finding container 5f971e976417078f31f8a447d04d6c8bfd2dfea7ff15b22ea37ce5c330931462: Status 404 returned error can't find the container with id 5f971e976417078f31f8a447d04d6c8bfd2dfea7ff15b22ea37ce5c330931462
Oct 31 03:31:36.314431 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:36.314393    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98" event={"ID":"dc77befa-5d3a-421f-8076-25786aac3569","Type":"ContainerStarted","Data":"5f971e976417078f31f8a447d04d6c8bfd2dfea7ff15b22ea37ce5c330931462"}
Oct 31 03:31:37.323788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:37.323734    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98" event={"ID":"dc77befa-5d3a-421f-8076-25786aac3569","Type":"ContainerStarted","Data":"3d92de3f7b6ec743c82c4043a41ea618b808c36ad31c7674f21a5a1859167254"}
Oct 31 03:31:37.324237 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:37.323819    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:37.435625 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:37.435500    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98" podStartSLOduration=2.462321911 podStartE2EDuration="3.435482301s" totalImagesPullingTime="973.16039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:37.432244738 +0000 UTC m=+1438.992724468" watchObservedRunningTime="2026-10-31 03:31:37.435482301 +0000 UTC m=+1438.995962031"
Oct 31 03:31:43.341424 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:31:43.341394    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:31:46.176210 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:31:46.176126    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:32:11.038183 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.038142    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"]
Oct 31 03:32:11.043045 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.043019    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.088095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.088054    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.088095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.088100    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.088334 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.088205    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515331d3-50e0-4935-a5ae-860f67116f6e-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.088334 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.088276    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.088401 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.088380    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.088434 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.088419    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsgr8\" (UniqueName: \"kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-kube-api-access-qsgr8\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.088470 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.088440    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.088572 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.088557    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.088609 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.088600    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515331d3-50e0-4935-a5ae-860f67116f6e-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.143108 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.143070    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"]
Oct 31 03:32:11.189489 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.189441    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.189721 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.189507    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515331d3-50e0-4935-a5ae-860f67116f6e-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.189721 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.189573    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.189721 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.189591    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.189721 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.189646    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515331d3-50e0-4935-a5ae-860f67116f6e-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.189721 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.189670    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.190057 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.189757    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.190057 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.189817    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsgr8\" (UniqueName: \"kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-kube-api-access-qsgr8\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.190057 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.189845    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.190057 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.190004    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.190261 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.190088    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.190261 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.190247    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.190368 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.190259    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.190579 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.190556    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515331d3-50e0-4935-a5ae-860f67116f6e-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.192161 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.192135    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.192291 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.192225    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515331d3-50e0-4935-a5ae-860f67116f6e-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.243913 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.243856    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.244073 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.243927    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsgr8\" (UniqueName: \"kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-kube-api-access-qsgr8\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.358169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.358138    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:11.779441 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.779404    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"]
Oct 31 03:32:11.781288 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:32:11.781261    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515331d3_50e0_4935_a5ae_860f67116f6e.slice/crio-764b89163c724e7fb542c6e329cca14a4ff0df8a8696a337d9b929340c254d7f WatchSource:0}: Error finding container 764b89163c724e7fb542c6e329cca14a4ff0df8a8696a337d9b929340c254d7f: Status 404 returned error can't find the container with id 764b89163c724e7fb542c6e329cca14a4ff0df8a8696a337d9b929340c254d7f
Oct 31 03:32:11.783907 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.783847    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:32:11.783998 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.783945    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:32:11.783998 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:11.783978    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:32:12.575222 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:12.575186    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt" event={"ID":"515331d3-50e0-4935-a5ae-860f67116f6e","Type":"ContainerStarted","Data":"0e8754f32c05492bdd76ab4d194955cc44f2901cee8a745c3bc4314978ebaaf4"}
Oct 31 03:32:12.575744 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:12.575247    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt" event={"ID":"515331d3-50e0-4935-a5ae-860f67116f6e","Type":"ContainerStarted","Data":"764b89163c724e7fb542c6e329cca14a4ff0df8a8696a337d9b929340c254d7f"}
Oct 31 03:32:12.687941 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:12.687872    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt" podStartSLOduration=2.6878555349999997 podStartE2EDuration="2.687855535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:12.686018522 +0000 UTC m=+1474.246498273" watchObservedRunningTime="2026-10-31 03:32:12.687855535 +0000 UTC m=+1474.248335335"
Oct 31 03:32:13.363915 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:13.361337    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:13.373394 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:13.373353    5415 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 03:32:13.373599 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:13.373426    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt" podUID="515331d3-50e0-4935-a5ae-860f67116f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 03:32:14.371951 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:14.364390    5415 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 03:32:14.371951 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:14.364463    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt" podUID="515331d3-50e0-4935-a5ae-860f67116f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 03:32:15.362324 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:15.362281    5415 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 03:32:15.362669 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:15.362632    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt" podUID="515331d3-50e0-4935-a5ae-860f67116f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 03:32:16.198935 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:32:16.198894    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:32:16.372132 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:16.372061    5415 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 03:32:16.372132 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:16.372129    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt" podUID="515331d3-50e0-4935-a5ae-860f67116f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 03:32:17.377036 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:17.376993    5415 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 03:32:17.377642 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:17.377076    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt" podUID="515331d3-50e0-4935-a5ae-860f67116f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 03:32:18.359291 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:18.359247    5415 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 03:32:18.359488 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:18.359335    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt" podUID="515331d3-50e0-4935-a5ae-860f67116f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 03:32:19.364082 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:19.364052    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:19.364480 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:19.364349    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:19.365277 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:19.365257    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:32:27.638578 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.638539    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"]
Oct 31 03:32:27.643483 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.643456    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:27.693207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.693175    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5effde-serving-cert\""
Oct 31 03:32:27.741497 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.741432    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-maas-api-tls\") pod \"maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:27.741715 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.741537    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-metrics-certs\") pod \"maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:27.741825 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.741802    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49wpk\" (UniqueName: \"kubernetes.io/projected/8989cbdb-731f-480c-9821-d9532967a613-kube-api-access-49wpk\") pod \"maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:27.741984 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.741965    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8989cbdb-731f-480c-9821-d9532967a613-openshift-service-ca\") pod \"maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:27.743476 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.743451    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"]
Oct 31 03:32:27.843313 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.843258    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49wpk\" (UniqueName: \"kubernetes.io/projected/8989cbdb-731f-480c-9821-d9532967a613-kube-api-access-49wpk\") pod \"maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:27.843523 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.843370    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8989cbdb-731f-480c-9821-d9532967a613-openshift-service-ca\") pod \"maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:27.843523 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.843496    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-maas-api-tls\") pod \"maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:27.843523 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.843517    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-metrics-certs\") pod \"maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:27.843695 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:32:27.843672    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-serving-cert: secret "maas-api-e2e-api-a-5effde-serving-cert" not found
Oct 31 03:32:27.843762 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:32:27.843751    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-maas-api-tls podName:8989cbdb-731f-480c-9821-d9532967a613 nodeName:}" failed. No retries permitted until 2026-10-31 03:32:28.343729473 +0000 UTC m=+1489.904209182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-maas-api-tls") pod "maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp" (UID: "8989cbdb-731f-480c-9821-d9532967a613") : secret "maas-api-e2e-api-a-5effde-serving-cert" not found
Oct 31 03:32:27.844151 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.844126    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8989cbdb-731f-480c-9821-d9532967a613-openshift-service-ca\") pod \"maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:27.846297 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.846276    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-metrics-certs\") pod \"maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:27.900835 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:27.900755    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49wpk\" (UniqueName: \"kubernetes.io/projected/8989cbdb-731f-480c-9821-d9532967a613-kube-api-access-49wpk\") pod \"maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:28.351011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:28.350970    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-maas-api-tls\") pod \"maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:28.353540 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:28.353507    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-maas-api-tls\") pod \"maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:28.558757 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:28.558718    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:28.935075 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:28.935042    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"]
Oct 31 03:32:28.936012 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:32:28.935975    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8989cbdb_731f_480c_9821_d9532967a613.slice/crio-9eab8868e62593771a83a47735a3ce2cf82ca2f4abdc02628bd9dc6b4f882d58 WatchSource:0}: Error finding container 9eab8868e62593771a83a47735a3ce2cf82ca2f4abdc02628bd9dc6b4f882d58: Status 404 returned error can't find the container with id 9eab8868e62593771a83a47735a3ce2cf82ca2f4abdc02628bd9dc6b4f882d58
Oct 31 03:32:29.683711 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:29.683675    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp" event={"ID":"8989cbdb-731f-480c-9821-d9532967a613","Type":"ContainerStarted","Data":"9eab8868e62593771a83a47735a3ce2cf82ca2f4abdc02628bd9dc6b4f882d58"}
Oct 31 03:32:30.691167 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:30.691118    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp" event={"ID":"8989cbdb-731f-480c-9821-d9532967a613","Type":"ContainerStarted","Data":"aeb92b7f832466cdb4f18f37d1eaf4bbb1fc692bfce00d8d97c1e80e30f5c0ec"}
Oct 31 03:32:30.691581 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:30.691204    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:30.802482 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:30.802416    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp" podStartSLOduration=2.916183524 podStartE2EDuration="3.802395495s" totalImagesPullingTime="886.211971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:30.80196347 +0000 UTC m=+1492.362443229" watchObservedRunningTime="2026-10-31 03:32:30.802395495 +0000 UTC m=+1492.362875228"
Oct 31 03:32:36.703270 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:36.703239    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:32:46.182823 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:32:46.182782    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:32:48.575296 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.575261    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"]
Oct 31 03:32:48.575775 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.575519    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx" podUID="bc0c59d3-274c-4ecf-a8de-453ce7b43012" containerName="maas-api" containerID="cri-o://cf4ea7cad25b06fa45e538655d28c5e9ede0154ae0dd00dd7330d4151cd93c62" gracePeriod=30
Oct 31 03:32:48.813768 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.813726    5415 generic.go:356] "Generic (PLEG): container finished" podID="bc0c59d3-274c-4ecf-a8de-453ce7b43012" containerID="cf4ea7cad25b06fa45e538655d28c5e9ede0154ae0dd00dd7330d4151cd93c62" exitCode=0
Oct 31 03:32:48.813990 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.813914    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx" event={"ID":"bc0c59d3-274c-4ecf-a8de-453ce7b43012","Type":"ContainerDied","Data":"cf4ea7cad25b06fa45e538655d28c5e9ede0154ae0dd00dd7330d4151cd93c62"}
Oct 31 03:32:48.898215 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.898182    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:32:48.987688 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.987646    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-metrics-certs\") pod \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") "
Oct 31 03:32:48.987688 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.987695    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-maas-api-tls\") pod \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") "
Oct 31 03:32:48.988040 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.987925    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc0c59d3-274c-4ecf-a8de-453ce7b43012-kube-api-access-9mdk4\" (UniqueName: \"kubernetes.io/projected/bc0c59d3-274c-4ecf-a8de-453ce7b43012-kube-api-access-9mdk4\") pod \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") "
Oct 31 03:32:48.988040 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.987956    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc0c59d3-274c-4ecf-a8de-453ce7b43012-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0c59d3-274c-4ecf-a8de-453ce7b43012-openshift-service-ca\") pod \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\" (UID: \"bc0c59d3-274c-4ecf-a8de-453ce7b43012\") "
Oct 31 03:32:48.988344 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.988313    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc0c59d3-274c-4ecf-a8de-453ce7b43012-openshift-service-ca" pod "bc0c59d3-274c-4ecf-a8de-453ce7b43012" (UID: "bc0c59d3-274c-4ecf-a8de-453ce7b43012"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:48.988614 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.988590    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0c59d3-274c-4ecf-a8de-453ce7b43012-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:48.990402 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.990375    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-metrics-certs" pod "bc0c59d3-274c-4ecf-a8de-453ce7b43012" (UID: "bc0c59d3-274c-4ecf-a8de-453ce7b43012"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:32:48.990517 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.990421    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-maas-api-tls" pod "bc0c59d3-274c-4ecf-a8de-453ce7b43012" (UID: "bc0c59d3-274c-4ecf-a8de-453ce7b43012"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:32:48.990517 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:48.990437    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc0c59d3-274c-4ecf-a8de-453ce7b43012-kube-api-access-9mdk4" pod "bc0c59d3-274c-4ecf-a8de-453ce7b43012" (UID: "bc0c59d3-274c-4ecf-a8de-453ce7b43012"). InnerVolumeSpecName "kube-api-access-9mdk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:49.089615 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:49.089575    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mdk4\" (UniqueName: \"kubernetes.io/projected/bc0c59d3-274c-4ecf-a8de-453ce7b43012-kube-api-access-9mdk4\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:49.089615 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:49.089609    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:49.089615 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:49.089619    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0c59d3-274c-4ecf-a8de-453ce7b43012-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:49.820999 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:49.820967    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"
Oct 31 03:32:49.821514 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:49.820999    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx" event={"ID":"bc0c59d3-274c-4ecf-a8de-453ce7b43012","Type":"ContainerDied","Data":"c1b3e2890566e8ffa73de38cb4392f08bca09bdf4e3c9bffb8f80d3041cc14df"}
Oct 31 03:32:49.821514 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:49.821073    5415 scope.go:122] "RemoveContainer" containerID="cf4ea7cad25b06fa45e538655d28c5e9ede0154ae0dd00dd7330d4151cd93c62"
Oct 31 03:32:49.983226 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:49.983177    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"]
Oct 31 03:32:50.028271 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:50.028237    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx"]
Oct 31 03:32:50.607252 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:50.607218    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc0c59d3-274c-4ecf-a8de-453ce7b43012" path="/var/lib/kubelet/pods/bc0c59d3-274c-4ecf-a8de-453ce7b43012/volumes"
Oct 31 03:32:51.812718 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:32:51.812662    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-h7nwx" podUID="bc0c59d3-274c-4ecf-a8de-453ce7b43012" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.67:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:33:04.415525 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:04.415486    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"]
Oct 31 03:33:04.416083 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:04.415743    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274" podUID="55b328fb-b191-4e90-92bf-671dae7a3f89" containerName="istio-proxy" containerID="cri-o://7757c20bc7ccede8a9a5bbcc559b2eecc35c7062f3ad3459a9047706929fb3e8" gracePeriod=30
Oct 31 03:33:04.461949 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:04.461893    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"]
Oct 31 03:33:04.462127 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:04.462106    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r" podUID="37159aa5-167a-4b63-a39c-86a341139168" containerName="istio-proxy" containerID="cri-o://e6afe4588d53ebaf27211854ca57f0a360c0853cf11aedae3f3ab2cd0cfe76f1" gracePeriod=30
Oct 31 03:33:06.661747 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:06.661707    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"]
Oct 31 03:33:06.662217 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:06.661929    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h" podUID="f1f4e579-566a-4e6d-b555-ff1ec9b121d5" containerName="maas-api" containerID="cri-o://c6cd359e71e08974532d38d8ce61a3ba77088b3e583023769062ab111233adf4" gracePeriod=30
Oct 31 03:33:06.934638 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:06.934549    5415 generic.go:356] "Generic (PLEG): container finished" podID="f1f4e579-566a-4e6d-b555-ff1ec9b121d5" containerID="c6cd359e71e08974532d38d8ce61a3ba77088b3e583023769062ab111233adf4" exitCode=0
Oct 31 03:33:06.934638 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:06.934623    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h" event={"ID":"f1f4e579-566a-4e6d-b555-ff1ec9b121d5","Type":"ContainerDied","Data":"c6cd359e71e08974532d38d8ce61a3ba77088b3e583023769062ab111233adf4"}
Oct 31 03:33:07.000454 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.000423    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:33:07.035243 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.035211    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-kube-api-access-q6hgw\" (UniqueName: \"kubernetes.io/projected/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-kube-api-access-q6hgw\") pod \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") "
Oct 31 03:33:07.035434 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.035278    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-metrics-certs\") pod \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") "
Oct 31 03:33:07.035434 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.035409    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-openshift-service-ca\") pod \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") "
Oct 31 03:33:07.035546 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.035511    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-maas-api-tls\") pod \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\" (UID: \"f1f4e579-566a-4e6d-b555-ff1ec9b121d5\") "
Oct 31 03:33:07.035927 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.035841    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-openshift-service-ca" pod "f1f4e579-566a-4e6d-b555-ff1ec9b121d5" (UID: "f1f4e579-566a-4e6d-b555-ff1ec9b121d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:07.036100 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.036081    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:07.037801 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.037762    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-maas-api-tls" pod "f1f4e579-566a-4e6d-b555-ff1ec9b121d5" (UID: "f1f4e579-566a-4e6d-b555-ff1ec9b121d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:33:07.037923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.037804    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-metrics-certs" pod "f1f4e579-566a-4e6d-b555-ff1ec9b121d5" (UID: "f1f4e579-566a-4e6d-b555-ff1ec9b121d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:33:07.037923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.037843    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-kube-api-access-q6hgw" pod "f1f4e579-566a-4e6d-b555-ff1ec9b121d5" (UID: "f1f4e579-566a-4e6d-b555-ff1ec9b121d5"). InnerVolumeSpecName "kube-api-access-q6hgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:07.137070 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.137035    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:07.137070 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.137070    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6hgw\" (UniqueName: \"kubernetes.io/projected/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-kube-api-access-q6hgw\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:07.137300 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.137085    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f4e579-566a-4e6d-b555-ff1ec9b121d5-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:07.943109 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.943079    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"
Oct 31 03:33:07.943109 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.943086    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h" event={"ID":"f1f4e579-566a-4e6d-b555-ff1ec9b121d5","Type":"ContainerDied","Data":"d3840136f67e4d7f66d2a995dbcf2f984fbc7de0e75027a9228b650f4ae2eb2a"}
Oct 31 03:33:07.943635 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:07.943160    5415 scope.go:122] "RemoveContainer" containerID="c6cd359e71e08974532d38d8ce61a3ba77088b3e583023769062ab111233adf4"
Oct 31 03:33:08.100348 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:08.100314    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"]
Oct 31 03:33:08.147454 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:08.147413    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a16786-8589945596-rdp9h"]
Oct 31 03:33:08.606906 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:08.606857    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f4e579-566a-4e6d-b555-ff1ec9b121d5" path="/var/lib/kubelet/pods/f1f4e579-566a-4e6d-b555-ff1ec9b121d5/volumes"
Oct 31 03:33:09.624230 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.624193    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 03:33:09.624660 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.624252    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r" podUID="37159aa5-167a-4b63-a39c-86a341139168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 03:33:09.758174 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.758141    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:33:09.780344 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.780314    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:33:09.892537 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.892499    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-kube-api-access-p9lmn\" (UniqueName: \"kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-kube-api-access-p9lmn\") pod \"37159aa5-167a-4b63-a39c-86a341139168\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") "
Oct 31 03:33:09.892716 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.892588    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-token\" (UniqueName: \"kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-token\") pod \"55b328fb-b191-4e90-92bf-671dae7a3f89\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") "
Oct 31 03:33:09.892716 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.892654    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-podinfo\") pod \"55b328fb-b191-4e90-92bf-671dae7a3f89\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") "
Oct 31 03:33:09.892806 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.892716    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-data\") pod \"37159aa5-167a-4b63-a39c-86a341139168\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") "
Oct 31 03:33:09.892806 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.892745    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37159aa5-167a-4b63-a39c-86a341139168-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37159aa5-167a-4b63-a39c-86a341139168-istiod-ca-cert\") pod \"37159aa5-167a-4b63-a39c-86a341139168\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") "
Oct 31 03:33:09.892806 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.892798    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b328fb-b191-4e90-92bf-671dae7a3f89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b328fb-b191-4e90-92bf-671dae7a3f89-istiod-ca-cert\") pod \"55b328fb-b191-4e90-92bf-671dae7a3f89\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") "
Oct 31 03:33:09.892972 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.892832    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-envoy\") pod \"55b328fb-b191-4e90-92bf-671dae7a3f89\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") "
Oct 31 03:33:09.892972 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.892943    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-credential-socket\") pod \"37159aa5-167a-4b63-a39c-86a341139168\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") "
Oct 31 03:33:09.893075 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893053    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-socket\") pod \"37159aa5-167a-4b63-a39c-86a341139168\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") "
Oct 31 03:33:09.893133 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893107    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-credential-socket\") pod \"55b328fb-b191-4e90-92bf-671dae7a3f89\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") "
Oct 31 03:33:09.893193 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893134    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-socket\") pod \"55b328fb-b191-4e90-92bf-671dae7a3f89\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") "
Oct 31 03:33:09.893193 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893166    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37159aa5-167a-4b63-a39c-86a341139168-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37159aa5-167a-4b63-a39c-86a341139168-istio-podinfo\") pod \"37159aa5-167a-4b63-a39c-86a341139168\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") "
Oct 31 03:33:09.893299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893203    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b328fb-b191-4e90-92bf-671dae7a3f89-istiod-ca-cert" pod "55b328fb-b191-4e90-92bf-671dae7a3f89" (UID: "55b328fb-b191-4e90-92bf-671dae7a3f89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:09.893299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893230    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-data\") pod \"55b328fb-b191-4e90-92bf-671dae7a3f89\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") "
Oct 31 03:33:09.893299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893216    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37159aa5-167a-4b63-a39c-86a341139168-istiod-ca-cert" pod "37159aa5-167a-4b63-a39c-86a341139168" (UID: "37159aa5-167a-4b63-a39c-86a341139168"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:09.893299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893275    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-istio-token\" (UniqueName: \"kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-istio-token\") pod \"37159aa5-167a-4b63-a39c-86a341139168\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") "
Oct 31 03:33:09.893505 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893310    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-envoy\") pod \"37159aa5-167a-4b63-a39c-86a341139168\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") "
Oct 31 03:33:09.893505 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893383    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-certs\") pod \"37159aa5-167a-4b63-a39c-86a341139168\" (UID: \"37159aa5-167a-4b63-a39c-86a341139168\") "
Oct 31 03:33:09.893505 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893387    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-data" pod "37159aa5-167a-4b63-a39c-86a341139168" (UID: "37159aa5-167a-4b63-a39c-86a341139168"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.893505 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893425    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-certs\") pod \"55b328fb-b191-4e90-92bf-671dae7a3f89\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") "
Oct 31 03:33:09.893705 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893527    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-credential-socket" pod "37159aa5-167a-4b63-a39c-86a341139168" (UID: "37159aa5-167a-4b63-a39c-86a341139168"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.893705 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893527    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-credential-socket" pod "55b328fb-b191-4e90-92bf-671dae7a3f89" (UID: "55b328fb-b191-4e90-92bf-671dae7a3f89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.893705 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893574    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-kube-api-access-5v8cc\" (UniqueName: \"kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-kube-api-access-5v8cc\") pod \"55b328fb-b191-4e90-92bf-671dae7a3f89\" (UID: \"55b328fb-b191-4e90-92bf-671dae7a3f89\") "
Oct 31 03:33:09.893705 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893646    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-certs" pod "55b328fb-b191-4e90-92bf-671dae7a3f89" (UID: "55b328fb-b191-4e90-92bf-671dae7a3f89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.893921 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893730    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-socket" pod "37159aa5-167a-4b63-a39c-86a341139168" (UID: "37159aa5-167a-4b63-a39c-86a341139168"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.893921 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893739    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-data" pod "55b328fb-b191-4e90-92bf-671dae7a3f89" (UID: "55b328fb-b191-4e90-92bf-671dae7a3f89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.894037 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.893963    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-socket" pod "55b328fb-b191-4e90-92bf-671dae7a3f89" (UID: "55b328fb-b191-4e90-92bf-671dae7a3f89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.894556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.894284    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.894556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.894309    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37159aa5-167a-4b63-a39c-86a341139168-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.894556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.894325    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b328fb-b191-4e90-92bf-671dae7a3f89-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.894556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.894340    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.894556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.894354    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.894556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.894369    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.894556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.894383    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.894556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.894397    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.894556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.894482    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-certs" pod "37159aa5-167a-4b63-a39c-86a341139168" (UID: "37159aa5-167a-4b63-a39c-86a341139168"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.894556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.894505    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.895400 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.895375    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-podinfo" pod "55b328fb-b191-4e90-92bf-671dae7a3f89" (UID: "55b328fb-b191-4e90-92bf-671dae7a3f89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:09.895614 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.895585    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-token" pod "55b328fb-b191-4e90-92bf-671dae7a3f89" (UID: "55b328fb-b191-4e90-92bf-671dae7a3f89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:09.895775 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.895753    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-kube-api-access-p9lmn" pod "37159aa5-167a-4b63-a39c-86a341139168" (UID: "37159aa5-167a-4b63-a39c-86a341139168"). InnerVolumeSpecName "kube-api-access-p9lmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:09.896000 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.895972    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-istio-token" pod "37159aa5-167a-4b63-a39c-86a341139168" (UID: "37159aa5-167a-4b63-a39c-86a341139168"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:09.896413 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.896235    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37159aa5-167a-4b63-a39c-86a341139168-istio-podinfo" pod "37159aa5-167a-4b63-a39c-86a341139168" (UID: "37159aa5-167a-4b63-a39c-86a341139168"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:09.896413 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.896340    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-kube-api-access-5v8cc" pod "55b328fb-b191-4e90-92bf-671dae7a3f89" (UID: "55b328fb-b191-4e90-92bf-671dae7a3f89"). InnerVolumeSpecName "kube-api-access-5v8cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:09.897025 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.896996    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-envoy" pod "55b328fb-b191-4e90-92bf-671dae7a3f89" (UID: "55b328fb-b191-4e90-92bf-671dae7a3f89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.897238 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.897220    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-envoy" pod "37159aa5-167a-4b63-a39c-86a341139168" (UID: "37159aa5-167a-4b63-a39c-86a341139168"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.960145 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.960040    5415 generic.go:356] "Generic (PLEG): container finished" podID="55b328fb-b191-4e90-92bf-671dae7a3f89" containerID="7757c20bc7ccede8a9a5bbcc559b2eecc35c7062f3ad3459a9047706929fb3e8" exitCode=0
Oct 31 03:33:09.960145 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.960095    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"
Oct 31 03:33:09.960145 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.960106    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274" event={"ID":"55b328fb-b191-4e90-92bf-671dae7a3f89","Type":"ContainerDied","Data":"7757c20bc7ccede8a9a5bbcc559b2eecc35c7062f3ad3459a9047706929fb3e8"}
Oct 31 03:33:09.960430 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.960173    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274" event={"ID":"55b328fb-b191-4e90-92bf-671dae7a3f89","Type":"ContainerDied","Data":"b4150f31035518288d2cd7bfaca8e7e21a6d77c3024e56e8ca5fd8672e7fb953"}
Oct 31 03:33:09.960430 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.960203    5415 scope.go:122] "RemoveContainer" containerID="7757c20bc7ccede8a9a5bbcc559b2eecc35c7062f3ad3459a9047706929fb3e8"
Oct 31 03:33:09.962398 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.962367    5415 generic.go:356] "Generic (PLEG): container finished" podID="37159aa5-167a-4b63-a39c-86a341139168" containerID="e6afe4588d53ebaf27211854ca57f0a360c0853cf11aedae3f3ab2cd0cfe76f1" exitCode=0
Oct 31 03:33:09.962545 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.962431    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r" event={"ID":"37159aa5-167a-4b63-a39c-86a341139168","Type":"ContainerDied","Data":"e6afe4588d53ebaf27211854ca57f0a360c0853cf11aedae3f3ab2cd0cfe76f1"}
Oct 31 03:33:09.962545 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.962476    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r" event={"ID":"37159aa5-167a-4b63-a39c-86a341139168","Type":"ContainerDied","Data":"006ff97c74eb8d1ba9c99f101317046fb427e9f7c0c4d5c60341d20dd828dada"}
Oct 31 03:33:09.962672 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.962552    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"
Oct 31 03:33:09.980473 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.980451    5415 scope.go:122] "RemoveContainer" containerID="7757c20bc7ccede8a9a5bbcc559b2eecc35c7062f3ad3459a9047706929fb3e8"
Oct 31 03:33:09.980846 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:33:09.980826    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7757c20bc7ccede8a9a5bbcc559b2eecc35c7062f3ad3459a9047706929fb3e8\": container with ID starting with 7757c20bc7ccede8a9a5bbcc559b2eecc35c7062f3ad3459a9047706929fb3e8 not found: ID does not exist" containerID="7757c20bc7ccede8a9a5bbcc559b2eecc35c7062f3ad3459a9047706929fb3e8"
Oct 31 03:33:09.980916 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.980862    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7757c20bc7ccede8a9a5bbcc559b2eecc35c7062f3ad3459a9047706929fb3e8"} err="failed to get container status \"7757c20bc7ccede8a9a5bbcc559b2eecc35c7062f3ad3459a9047706929fb3e8\": rpc error: code = NotFound desc = could not find container \"7757c20bc7ccede8a9a5bbcc559b2eecc35c7062f3ad3459a9047706929fb3e8\": container with ID starting with 7757c20bc7ccede8a9a5bbcc559b2eecc35c7062f3ad3459a9047706929fb3e8 not found: ID does not exist"
Oct 31 03:33:09.980963 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.980918    5415 scope.go:122] "RemoveContainer" containerID="e6afe4588d53ebaf27211854ca57f0a360c0853cf11aedae3f3ab2cd0cfe76f1"
Oct 31 03:33:09.990589 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.990565    5415 scope.go:122] "RemoveContainer" containerID="e6afe4588d53ebaf27211854ca57f0a360c0853cf11aedae3f3ab2cd0cfe76f1"
Oct 31 03:33:09.990936 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:33:09.990914    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6afe4588d53ebaf27211854ca57f0a360c0853cf11aedae3f3ab2cd0cfe76f1\": container with ID starting with e6afe4588d53ebaf27211854ca57f0a360c0853cf11aedae3f3ab2cd0cfe76f1 not found: ID does not exist" containerID="e6afe4588d53ebaf27211854ca57f0a360c0853cf11aedae3f3ab2cd0cfe76f1"
Oct 31 03:33:09.991011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.990946    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6afe4588d53ebaf27211854ca57f0a360c0853cf11aedae3f3ab2cd0cfe76f1"} err="failed to get container status \"e6afe4588d53ebaf27211854ca57f0a360c0853cf11aedae3f3ab2cd0cfe76f1\": rpc error: code = NotFound desc = could not find container \"e6afe4588d53ebaf27211854ca57f0a360c0853cf11aedae3f3ab2cd0cfe76f1\": container with ID starting with e6afe4588d53ebaf27211854ca57f0a360c0853cf11aedae3f3ab2cd0cfe76f1 not found: ID does not exist"
Oct 31 03:33:09.995704 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.995680    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.995704 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.995706    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.995842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.995716    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37159aa5-167a-4b63-a39c-86a341139168-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.995842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.995727    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8cc\" (UniqueName: \"kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-kube-api-access-5v8cc\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.995842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.995740    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9lmn\" (UniqueName: \"kubernetes.io/projected/37159aa5-167a-4b63-a39c-86a341139168-kube-api-access-p9lmn\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.995842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.995749    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.995842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.995759    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.995842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.995768    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b328fb-b191-4e90-92bf-671dae7a3f89-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.995842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:09.995776    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37159aa5-167a-4b63-a39c-86a341139168-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:10.138410 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:10.138371    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"]
Oct 31 03:33:10.142334 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:10.142306    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-4qc6r"]
Oct 31 03:33:10.245762 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:10.245661    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"]
Oct 31 03:33:10.251203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:10.251170    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-nf274"]
Oct 31 03:33:10.606608 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:10.606569    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37159aa5-167a-4b63-a39c-86a341139168" path="/var/lib/kubelet/pods/37159aa5-167a-4b63-a39c-86a341139168/volumes"
Oct 31 03:33:10.607111 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:10.607095    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b328fb-b191-4e90-92bf-671dae7a3f89" path="/var/lib/kubelet/pods/55b328fb-b191-4e90-92bf-671dae7a3f89/volumes"
Oct 31 03:33:11.422552 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:11.422509    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"]
Oct 31 03:33:11.422964 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:11.422727    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr" podUID="0f3173f7-3b7d-4765-94b9-506ca9765abf" containerName="istio-proxy" containerID="cri-o://b776103e4294dcf23bee9f88057592a8117ad30ae534eb0d3a79d8c36c27717f" gracePeriod=30
Oct 31 03:33:16.177079 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:33:16.176981    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:33:16.724824 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.724791    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:33:16.792088 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.792045    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-certs\") pod \"0f3173f7-3b7d-4765-94b9-506ca9765abf\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") "
Oct 31 03:33:16.792088 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.792096    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-credential-socket\") pod \"0f3173f7-3b7d-4765-94b9-506ca9765abf\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") "
Oct 31 03:33:16.792352 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.792181    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-token\" (UniqueName: \"kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-token\") pod \"0f3173f7-3b7d-4765-94b9-506ca9765abf\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") "
Oct 31 03:33:16.792352 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.792229    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-data\") pod \"0f3173f7-3b7d-4765-94b9-506ca9765abf\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") "
Oct 31 03:33:16.792352 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.792275    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-kube-api-access-vxnwf\" (UniqueName: \"kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-kube-api-access-vxnwf\") pod \"0f3173f7-3b7d-4765-94b9-506ca9765abf\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") "
Oct 31 03:33:16.792513 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.792366    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-socket\") pod \"0f3173f7-3b7d-4765-94b9-506ca9765abf\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") "
Oct 31 03:33:16.792513 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.792364    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-certs" pod "0f3173f7-3b7d-4765-94b9-506ca9765abf" (UID: "0f3173f7-3b7d-4765-94b9-506ca9765abf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:16.792513 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.792397    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-envoy\") pod \"0f3173f7-3b7d-4765-94b9-506ca9765abf\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") "
Oct 31 03:33:16.792513 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.792451    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-credential-socket" pod "0f3173f7-3b7d-4765-94b9-506ca9765abf" (UID: "0f3173f7-3b7d-4765-94b9-506ca9765abf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:16.792513 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.792464    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-podinfo\") pod \"0f3173f7-3b7d-4765-94b9-506ca9765abf\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") "
Oct 31 03:33:16.792767 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.792535    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3173f7-3b7d-4765-94b9-506ca9765abf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3173f7-3b7d-4765-94b9-506ca9765abf-istiod-ca-cert\") pod \"0f3173f7-3b7d-4765-94b9-506ca9765abf\" (UID: \"0f3173f7-3b7d-4765-94b9-506ca9765abf\") "
Oct 31 03:33:16.792767 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.792575    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-data" pod "0f3173f7-3b7d-4765-94b9-506ca9765abf" (UID: "0f3173f7-3b7d-4765-94b9-506ca9765abf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:16.792767 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.792614    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-socket" pod "0f3173f7-3b7d-4765-94b9-506ca9765abf" (UID: "0f3173f7-3b7d-4765-94b9-506ca9765abf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:16.793044 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.793021    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3173f7-3b7d-4765-94b9-506ca9765abf-istiod-ca-cert" pod "0f3173f7-3b7d-4765-94b9-506ca9765abf" (UID: "0f3173f7-3b7d-4765-94b9-506ca9765abf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:16.793235 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.793072    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:16.793235 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.793092    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:16.793235 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.793107    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:16.793235 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.793120    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:16.794630 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.794597    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-token" pod "0f3173f7-3b7d-4765-94b9-506ca9765abf" (UID: "0f3173f7-3b7d-4765-94b9-506ca9765abf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:16.794727 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.794659    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-kube-api-access-vxnwf" pod "0f3173f7-3b7d-4765-94b9-506ca9765abf" (UID: "0f3173f7-3b7d-4765-94b9-506ca9765abf"). InnerVolumeSpecName "kube-api-access-vxnwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:16.794911 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.794859    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-envoy" pod "0f3173f7-3b7d-4765-94b9-506ca9765abf" (UID: "0f3173f7-3b7d-4765-94b9-506ca9765abf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:16.794961 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.794909    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-podinfo" pod "0f3173f7-3b7d-4765-94b9-506ca9765abf" (UID: "0f3173f7-3b7d-4765-94b9-506ca9765abf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:16.894507 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.894464    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:16.894507 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.894502    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:16.894743 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.894518    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3173f7-3b7d-4765-94b9-506ca9765abf-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:16.894743 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.894531    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:16.894743 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:16.894544    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxnwf\" (UniqueName: \"kubernetes.io/projected/0f3173f7-3b7d-4765-94b9-506ca9765abf-kube-api-access-vxnwf\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.011397 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:17.011318    5415 generic.go:356] "Generic (PLEG): container finished" podID="0f3173f7-3b7d-4765-94b9-506ca9765abf" containerID="b776103e4294dcf23bee9f88057592a8117ad30ae534eb0d3a79d8c36c27717f" exitCode=0
Oct 31 03:33:17.011397 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:17.011362    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"
Oct 31 03:33:17.011397 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:17.011380    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr" event={"ID":"0f3173f7-3b7d-4765-94b9-506ca9765abf","Type":"ContainerDied","Data":"b776103e4294dcf23bee9f88057592a8117ad30ae534eb0d3a79d8c36c27717f"}
Oct 31 03:33:17.011666 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:17.011424    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr" event={"ID":"0f3173f7-3b7d-4765-94b9-506ca9765abf","Type":"ContainerDied","Data":"4b4d87fe749dea0a54794e56675d58f68f23e49888bbe728ac2b7ac2666cec9d"}
Oct 31 03:33:17.011666 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:17.011448    5415 scope.go:122] "RemoveContainer" containerID="b776103e4294dcf23bee9f88057592a8117ad30ae534eb0d3a79d8c36c27717f"
Oct 31 03:33:17.022232 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:17.022212    5415 scope.go:122] "RemoveContainer" containerID="b776103e4294dcf23bee9f88057592a8117ad30ae534eb0d3a79d8c36c27717f"
Oct 31 03:33:17.022541 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:33:17.022518    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b776103e4294dcf23bee9f88057592a8117ad30ae534eb0d3a79d8c36c27717f\": container with ID starting with b776103e4294dcf23bee9f88057592a8117ad30ae534eb0d3a79d8c36c27717f not found: ID does not exist" containerID="b776103e4294dcf23bee9f88057592a8117ad30ae534eb0d3a79d8c36c27717f"
Oct 31 03:33:17.022592 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:17.022552    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b776103e4294dcf23bee9f88057592a8117ad30ae534eb0d3a79d8c36c27717f"} err="failed to get container status \"b776103e4294dcf23bee9f88057592a8117ad30ae534eb0d3a79d8c36c27717f\": rpc error: code = NotFound desc = could not find container \"b776103e4294dcf23bee9f88057592a8117ad30ae534eb0d3a79d8c36c27717f\": container with ID starting with b776103e4294dcf23bee9f88057592a8117ad30ae534eb0d3a79d8c36c27717f not found: ID does not exist"
Oct 31 03:33:17.192062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:17.192028    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"]
Oct 31 03:33:17.194650 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:17.194622    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-p87gr"]
Oct 31 03:33:18.607101 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:18.607069    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3173f7-3b7d-4765-94b9-506ca9765abf" path="/var/lib/kubelet/pods/0f3173f7-3b7d-4765-94b9-506ca9765abf/volumes"
Oct 31 03:33:29.817735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:29.817639    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"]
Oct 31 03:33:29.818156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:29.817944    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98" podUID="dc77befa-5d3a-421f-8076-25786aac3569" containerName="maas-api" containerID="cri-o://3d92de3f7b6ec743c82c4043a41ea618b808c36ad31c7674f21a5a1859167254" gracePeriod=30
Oct 31 03:33:30.107644 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.107619    5415 generic.go:356] "Generic (PLEG): container finished" podID="dc77befa-5d3a-421f-8076-25786aac3569" containerID="3d92de3f7b6ec743c82c4043a41ea618b808c36ad31c7674f21a5a1859167254" exitCode=0
Oct 31 03:33:30.107718 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.107690    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98" event={"ID":"dc77befa-5d3a-421f-8076-25786aac3569","Type":"ContainerDied","Data":"3d92de3f7b6ec743c82c4043a41ea618b808c36ad31c7674f21a5a1859167254"}
Oct 31 03:33:30.148753 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.148728    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:33:30.272589 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.272339    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-metrics-certs\") pod \"dc77befa-5d3a-421f-8076-25786aac3569\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") "
Oct 31 03:33:30.272589 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.272418    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc77befa-5d3a-421f-8076-25786aac3569-kube-api-access-2gg9n\" (UniqueName: \"kubernetes.io/projected/dc77befa-5d3a-421f-8076-25786aac3569-kube-api-access-2gg9n\") pod \"dc77befa-5d3a-421f-8076-25786aac3569\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") "
Oct 31 03:33:30.272589 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.272473    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc77befa-5d3a-421f-8076-25786aac3569-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc77befa-5d3a-421f-8076-25786aac3569-openshift-service-ca\") pod \"dc77befa-5d3a-421f-8076-25786aac3569\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") "
Oct 31 03:33:30.272589 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.272519    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-maas-api-tls\") pod \"dc77befa-5d3a-421f-8076-25786aac3569\" (UID: \"dc77befa-5d3a-421f-8076-25786aac3569\") "
Oct 31 03:33:30.274311 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.274195    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc77befa-5d3a-421f-8076-25786aac3569-openshift-service-ca" pod "dc77befa-5d3a-421f-8076-25786aac3569" (UID: "dc77befa-5d3a-421f-8076-25786aac3569"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:30.277225 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.276056    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc77befa-5d3a-421f-8076-25786aac3569-kube-api-access-2gg9n" pod "dc77befa-5d3a-421f-8076-25786aac3569" (UID: "dc77befa-5d3a-421f-8076-25786aac3569"). InnerVolumeSpecName "kube-api-access-2gg9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:30.295905 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.278849    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-metrics-certs" pod "dc77befa-5d3a-421f-8076-25786aac3569" (UID: "dc77befa-5d3a-421f-8076-25786aac3569"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:33:30.295905 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.278894    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-maas-api-tls" pod "dc77befa-5d3a-421f-8076-25786aac3569" (UID: "dc77befa-5d3a-421f-8076-25786aac3569"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:33:30.374474 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.374380    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gg9n\" (UniqueName: \"kubernetes.io/projected/dc77befa-5d3a-421f-8076-25786aac3569-kube-api-access-2gg9n\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:30.374474 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.374415    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc77befa-5d3a-421f-8076-25786aac3569-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:30.374474 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.374426    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:30.374474 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:30.374434    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc77befa-5d3a-421f-8076-25786aac3569-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:31.114814 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:31.114780    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"
Oct 31 03:33:31.114814 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:31.114806    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98" event={"ID":"dc77befa-5d3a-421f-8076-25786aac3569","Type":"ContainerDied","Data":"5f971e976417078f31f8a447d04d6c8bfd2dfea7ff15b22ea37ce5c330931462"}
Oct 31 03:33:31.115300 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:31.114870    5415 scope.go:122] "RemoveContainer" containerID="3d92de3f7b6ec743c82c4043a41ea618b808c36ad31c7674f21a5a1859167254"
Oct 31 03:33:31.279144 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:31.279105    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"]
Oct 31 03:33:31.323865 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:31.323824    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-070d07-78bc48c648-wlj98"]
Oct 31 03:33:32.606143 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:33:32.606107    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc77befa-5d3a-421f-8076-25786aac3569" path="/var/lib/kubelet/pods/dc77befa-5d3a-421f-8076-25786aac3569/volumes"
Oct 31 03:33:46.179127 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:33:46.179081    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:34:03.352820 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.352780    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"]
Oct 31 03:34:03.353434 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.353062    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8" podUID="528321a5-da5c-415b-b3f2-4030a4945aa2" containerName="maas-api" containerID="cri-o://cb7d42a59b615787ae1eed13d226a79b472121f2967b3b951aa7987a5089b539" gracePeriod=30
Oct 31 03:34:03.692164 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.692130    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:34:03.792934 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.792899    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-maas-api-tls\") pod \"528321a5-da5c-415b-b3f2-4030a4945aa2\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") "
Oct 31 03:34:03.793207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.793181    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-metrics-certs\") pod \"528321a5-da5c-415b-b3f2-4030a4945aa2\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") "
Oct 31 03:34:03.793553 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.793532    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528321a5-da5c-415b-b3f2-4030a4945aa2-kube-api-access-4pgzj\" (UniqueName: \"kubernetes.io/projected/528321a5-da5c-415b-b3f2-4030a4945aa2-kube-api-access-4pgzj\") pod \"528321a5-da5c-415b-b3f2-4030a4945aa2\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") "
Oct 31 03:34:03.793766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.793751    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528321a5-da5c-415b-b3f2-4030a4945aa2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528321a5-da5c-415b-b3f2-4030a4945aa2-openshift-service-ca\") pod \"528321a5-da5c-415b-b3f2-4030a4945aa2\" (UID: \"528321a5-da5c-415b-b3f2-4030a4945aa2\") "
Oct 31 03:34:03.794151 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.794125    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528321a5-da5c-415b-b3f2-4030a4945aa2-openshift-service-ca" pod "528321a5-da5c-415b-b3f2-4030a4945aa2" (UID: "528321a5-da5c-415b-b3f2-4030a4945aa2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:34:03.794570 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.794551    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528321a5-da5c-415b-b3f2-4030a4945aa2-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:03.796172 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.796140    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-maas-api-tls" pod "528321a5-da5c-415b-b3f2-4030a4945aa2" (UID: "528321a5-da5c-415b-b3f2-4030a4945aa2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:34:03.798140 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.798112    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528321a5-da5c-415b-b3f2-4030a4945aa2-kube-api-access-4pgzj" pod "528321a5-da5c-415b-b3f2-4030a4945aa2" (UID: "528321a5-da5c-415b-b3f2-4030a4945aa2"). InnerVolumeSpecName "kube-api-access-4pgzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:03.798265 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.798252    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-metrics-certs" pod "528321a5-da5c-415b-b3f2-4030a4945aa2" (UID: "528321a5-da5c-415b-b3f2-4030a4945aa2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:34:03.895489 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.895450    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:03.895489 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.895483    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/528321a5-da5c-415b-b3f2-4030a4945aa2-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:03.895708 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:03.895516    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pgzj\" (UniqueName: \"kubernetes.io/projected/528321a5-da5c-415b-b3f2-4030a4945aa2-kube-api-access-4pgzj\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:04.369797 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:04.369749    5415 generic.go:356] "Generic (PLEG): container finished" podID="528321a5-da5c-415b-b3f2-4030a4945aa2" containerID="cb7d42a59b615787ae1eed13d226a79b472121f2967b3b951aa7987a5089b539" exitCode=0
Oct 31 03:34:04.370349 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:04.369843    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8" event={"ID":"528321a5-da5c-415b-b3f2-4030a4945aa2","Type":"ContainerDied","Data":"cb7d42a59b615787ae1eed13d226a79b472121f2967b3b951aa7987a5089b539"}
Oct 31 03:34:04.370349 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:04.369949    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8" event={"ID":"528321a5-da5c-415b-b3f2-4030a4945aa2","Type":"ContainerDied","Data":"5f99886a6dee55043eba3eb9fcd49883f11245a7d6e7171d3aa1082122a89224"}
Oct 31 03:34:04.370349 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:04.369981    5415 scope.go:122] "RemoveContainer" containerID="cb7d42a59b615787ae1eed13d226a79b472121f2967b3b951aa7987a5089b539"
Oct 31 03:34:04.370349 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:04.370185    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"
Oct 31 03:34:04.389933 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:04.389902    5415 scope.go:122] "RemoveContainer" containerID="cb7d42a59b615787ae1eed13d226a79b472121f2967b3b951aa7987a5089b539"
Oct 31 03:34:04.390377 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:34:04.390339    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7d42a59b615787ae1eed13d226a79b472121f2967b3b951aa7987a5089b539\": container with ID starting with cb7d42a59b615787ae1eed13d226a79b472121f2967b3b951aa7987a5089b539 not found: ID does not exist" containerID="cb7d42a59b615787ae1eed13d226a79b472121f2967b3b951aa7987a5089b539"
Oct 31 03:34:04.390489 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:04.390374    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7d42a59b615787ae1eed13d226a79b472121f2967b3b951aa7987a5089b539"} err="failed to get container status \"cb7d42a59b615787ae1eed13d226a79b472121f2967b3b951aa7987a5089b539\": rpc error: code = NotFound desc = could not find container \"cb7d42a59b615787ae1eed13d226a79b472121f2967b3b951aa7987a5089b539\": container with ID starting with cb7d42a59b615787ae1eed13d226a79b472121f2967b3b951aa7987a5089b539 not found: ID does not exist"
Oct 31 03:34:04.534227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:04.534190    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"]
Oct 31 03:34:04.580415 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:04.580376    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4fdeb-d7994d89d-b5gm8"]
Oct 31 03:34:04.606587 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:04.606542    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528321a5-da5c-415b-b3f2-4030a4945aa2" path="/var/lib/kubelet/pods/528321a5-da5c-415b-b3f2-4030a4945aa2/volumes"
Oct 31 03:34:16.177380 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:34:16.177337    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:34:46.176404 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:34:46.176322    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:34:49.030405 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.030366    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"]
Oct 31 03:34:49.031055 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031034    5415 cpu_manager.go:414] "containerMap: removing container" podUID="0f3173f7-3b7d-4765-94b9-506ca9765abf" containerName="istio-proxy"
Oct 31 03:34:49.031055 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031055    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3173f7-3b7d-4765-94b9-506ca9765abf" containerName="istio-proxy"
Oct 31 03:34:49.031214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031071    5415 cpu_manager.go:414] "containerMap: removing container" podUID="f1f4e579-566a-4e6d-b555-ff1ec9b121d5" containerName="maas-api"
Oct 31 03:34:49.031214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031076    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f4e579-566a-4e6d-b555-ff1ec9b121d5" containerName="maas-api"
Oct 31 03:34:49.031214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031088    5415 cpu_manager.go:414] "containerMap: removing container" podUID="bc0c59d3-274c-4ecf-a8de-453ce7b43012" containerName="maas-api"
Oct 31 03:34:49.031214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031093    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc0c59d3-274c-4ecf-a8de-453ce7b43012" containerName="maas-api"
Oct 31 03:34:49.031214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031115    5415 cpu_manager.go:414] "containerMap: removing container" podUID="55b328fb-b191-4e90-92bf-671dae7a3f89" containerName="istio-proxy"
Oct 31 03:34:49.031214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031120    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b328fb-b191-4e90-92bf-671dae7a3f89" containerName="istio-proxy"
Oct 31 03:34:49.031214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031128    5415 cpu_manager.go:414] "containerMap: removing container" podUID="528321a5-da5c-415b-b3f2-4030a4945aa2" containerName="maas-api"
Oct 31 03:34:49.031214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031133    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528321a5-da5c-415b-b3f2-4030a4945aa2" containerName="maas-api"
Oct 31 03:34:49.031214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031140    5415 cpu_manager.go:414] "containerMap: removing container" podUID="37159aa5-167a-4b63-a39c-86a341139168" containerName="istio-proxy"
Oct 31 03:34:49.031214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031144    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37159aa5-167a-4b63-a39c-86a341139168" containerName="istio-proxy"
Oct 31 03:34:49.031214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031151    5415 cpu_manager.go:414] "containerMap: removing container" podUID="dc77befa-5d3a-421f-8076-25786aac3569" containerName="maas-api"
Oct 31 03:34:49.031214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031156    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc77befa-5d3a-421f-8076-25786aac3569" containerName="maas-api"
Oct 31 03:34:49.031794 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031231    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="37159aa5-167a-4b63-a39c-86a341139168" containerName="istio-proxy"
Oct 31 03:34:49.031794 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031241    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc77befa-5d3a-421f-8076-25786aac3569" containerName="maas-api"
Oct 31 03:34:49.031794 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031250    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3173f7-3b7d-4765-94b9-506ca9765abf" containerName="istio-proxy"
Oct 31 03:34:49.031794 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031258    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f4e579-566a-4e6d-b555-ff1ec9b121d5" containerName="maas-api"
Oct 31 03:34:49.031794 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031263    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc0c59d3-274c-4ecf-a8de-453ce7b43012" containerName="maas-api"
Oct 31 03:34:49.031794 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031272    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b328fb-b191-4e90-92bf-671dae7a3f89" containerName="istio-proxy"
Oct 31 03:34:49.031794 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.031278    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="528321a5-da5c-415b-b3f2-4030a4945aa2" containerName="maas-api"
Oct 31 03:34:49.036290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.036254    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.086168 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.086128    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6b3a29ec-openshift-default-dockercfg-4ddsz\""
Oct 31 03:34:49.129142 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.129093    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.129363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.129157    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.129363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.129235    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f3f891-01ae-4bc1-845f-43da70087727-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.129363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.129334    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.129555 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.129398    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.129555 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.129424    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.129555 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.129441    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7tq2\" (UniqueName: \"kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-kube-api-access-k7tq2\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.129722 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.129596    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f3f891-01ae-4bc1-845f-43da70087727-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.129722 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.129640    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.231382 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.231339    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f3f891-01ae-4bc1-845f-43da70087727-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.231598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.231409    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.231598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.231465    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.231598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.231509    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.231598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.231534    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f3f891-01ae-4bc1-845f-43da70087727-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.231598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.231593    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.231856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.231670    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.231856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.231696    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.231856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.231721    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7tq2\" (UniqueName: \"kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-kube-api-access-k7tq2\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.232030 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.231910    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.232156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.232130    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.232156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.232147    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.232290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.232270    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f3f891-01ae-4bc1-845f-43da70087727-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.232424 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.232403    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.233852 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.233828    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.234277 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.234255    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f3f891-01ae-4bc1-845f-43da70087727-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.237027 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.237009    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"]
Oct 31 03:34:49.286100 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.286022    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.286414 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.286383    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7tq2\" (UniqueName: \"kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-kube-api-access-k7tq2\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.354323 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.354281    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:49.777445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.777414    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"]
Oct 31 03:34:49.779306 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:34:49.779259    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f3f891_01ae_4bc1_845f_43da70087727.slice/crio-bb70e332346ccd1243910670ed1fb6e5b5bab88a73f189f169cd1a19b3228acb WatchSource:0}: Error finding container bb70e332346ccd1243910670ed1fb6e5b5bab88a73f189f169cd1a19b3228acb: Status 404 returned error can't find the container with id bb70e332346ccd1243910670ed1fb6e5b5bab88a73f189f169cd1a19b3228acb
Oct 31 03:34:49.782219 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.782183    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:34:49.782322 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.782257    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:34:49.782322 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:49.782289    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:34:50.697745 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:50.697713    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s" event={"ID":"03f3f891-01ae-4bc1-845f-43da70087727","Type":"ContainerStarted","Data":"eb0919564d62b0cab6f98b2431ee4c7fde6874fdd766b02dec14ab0261ad1312"}
Oct 31 03:34:50.698193 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:50.697770    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s" event={"ID":"03f3f891-01ae-4bc1-845f-43da70087727","Type":"ContainerStarted","Data":"bb70e332346ccd1243910670ed1fb6e5b5bab88a73f189f169cd1a19b3228acb"}
Oct 31 03:34:50.873529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:50.873455    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s" podStartSLOduration=2.873436701 podStartE2EDuration="2.873436701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:50.87239238 +0000 UTC m=+1632.432872131" watchObservedRunningTime="2026-10-31 03:34:50.873436701 +0000 UTC m=+1632.433916433"
Oct 31 03:34:51.358902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:51.358788    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:51.359130 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:51.359031    5415 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 03:34:51.359130 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:51.359082    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s" podUID="03f3f891-01ae-4bc1-845f-43da70087727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 03:34:52.368902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:52.368410    5415 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 03:34:52.368902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:52.368476    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s" podUID="03f3f891-01ae-4bc1-845f-43da70087727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 03:34:53.368419 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:53.368375    5415 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 03:34:53.368720 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:53.368688    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s" podUID="03f3f891-01ae-4bc1-845f-43da70087727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 03:34:54.373051 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:54.373005    5415 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 03:34:54.373572 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:54.373085    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s" podUID="03f3f891-01ae-4bc1-845f-43da70087727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 03:34:55.376068 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:55.376031    5415 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 03:34:55.376553 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:55.376527    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s" podUID="03f3f891-01ae-4bc1-845f-43da70087727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 03:34:56.360120 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:56.360082    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:56.360649 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:56.360625    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:34:56.361432 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:34:56.361412    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:35:03.245545 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:03.245503    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"]
Oct 31 03:35:04.154373 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.154335    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"]
Oct 31 03:35:04.158188 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.158165    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.254327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.254279    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.254836 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.254378    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjck5\" (UniqueName: \"kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-kube-api-access-jjck5\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.254836 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.254465    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.254836 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.254624    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.254836 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.254678    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db75d372-8a01-458f-95f3-fe20a249d47f-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.254836 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.254747    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db75d372-8a01-458f-95f3-fe20a249d47f-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.255134 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.254929    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.255134 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.254980    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.255134 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.255065    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.359205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.356695    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db75d372-8a01-458f-95f3-fe20a249d47f-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.359205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.356809    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.359205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.356852    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.359205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.356904    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.359205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.356988    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.359205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.357013    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjck5\" (UniqueName: \"kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-kube-api-access-jjck5\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.359205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.357088    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.359205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.357187    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.359205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.357245    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db75d372-8a01-458f-95f3-fe20a249d47f-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.370754 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.367731    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"]
Oct 31 03:35:04.372204 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.370837    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.372204 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.371489    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db75d372-8a01-458f-95f3-fe20a249d47f-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.372204 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.371723    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.372497 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.372472    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.372891 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.372857    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.373657 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.373226    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.382505 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.382467    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db75d372-8a01-458f-95f3-fe20a249d47f-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.426207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.426124    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.426207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.426132    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjck5\" (UniqueName: \"kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-kube-api-access-jjck5\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.473211 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.473174    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:04.965771 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:35:04.965735    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb75d372_8a01_458f_95f3_fe20a249d47f.slice/crio-39ed4b533d0bf65b80cf978fe5a904ae255b4921ca1f03e4b9bdd4e4e6d5aa44 WatchSource:0}: Error finding container 39ed4b533d0bf65b80cf978fe5a904ae255b4921ca1f03e4b9bdd4e4e6d5aa44: Status 404 returned error can't find the container with id 39ed4b533d0bf65b80cf978fe5a904ae255b4921ca1f03e4b9bdd4e4e6d5aa44
Oct 31 03:35:04.968692 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.968654    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:35:04.968842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.968726    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:35:04.968842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:04.968759    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:35:05.032514 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:05.032488    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"]
Oct 31 03:35:05.799074 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:05.799035    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt" event={"ID":"db75d372-8a01-458f-95f3-fe20a249d47f","Type":"ContainerStarted","Data":"7c6cb106c88c8c1e59446bd6381b1aaea59be7316cc7f2adb4f18799de9e944d"}
Oct 31 03:35:05.799486 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:05.799124    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt" event={"ID":"db75d372-8a01-458f-95f3-fe20a249d47f","Type":"ContainerStarted","Data":"39ed4b533d0bf65b80cf978fe5a904ae255b4921ca1f03e4b9bdd4e4e6d5aa44"}
Oct 31 03:35:05.928449 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:05.928391    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt" podStartSLOduration=1.928376005 podStartE2EDuration="1.928376005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:05.926796837 +0000 UTC m=+1647.487276567" watchObservedRunningTime="2026-10-31 03:35:05.928376005 +0000 UTC m=+1647.488855736"
Oct 31 03:35:06.487901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:06.481923    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:06.487901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:06.482588    5415 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:35:06.487901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:06.482678    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt" podUID="db75d372-8a01-458f-95f3-fe20a249d47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 03:35:07.473901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:07.473577    5415 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:35:07.473901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:07.473651    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt" podUID="db75d372-8a01-458f-95f3-fe20a249d47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 03:35:08.474522 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:08.474481    5415 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:35:08.475064 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:08.474548    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt" podUID="db75d372-8a01-458f-95f3-fe20a249d47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 03:35:09.473709 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:09.473668    5415 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:35:09.473958 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:09.473733    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt" podUID="db75d372-8a01-458f-95f3-fe20a249d47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 03:35:10.477902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:10.474605    5415 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:35:10.477902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:10.474689    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt" podUID="db75d372-8a01-458f-95f3-fe20a249d47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 03:35:11.620969 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:11.620933    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:11.621737 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:11.621711    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:11.622697 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:11.622673    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:35:16.176606 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:35:16.176563    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:35:18.267018 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.266977    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp" podUID="3947643c-111e-4350-9749-47643517df5c" containerName="main" containerID="cri-o://e91cb74f9db26b45fdb5981cf6a4289746a7091dc9d9029b008915b25fc65e06" gracePeriod=45
Oct 31 03:35:18.534899 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.534804    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:35:18.710644 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.710613    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-model-cache\") pod \"3947643c-111e-4350-9749-47643517df5c\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") "
Oct 31 03:35:18.710644 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.710656    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3947643c-111e-4350-9749-47643517df5c-kube-api-access-8srnh\" (UniqueName: \"kubernetes.io/projected/3947643c-111e-4350-9749-47643517df5c-kube-api-access-8srnh\") pod \"3947643c-111e-4350-9749-47643517df5c\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") "
Oct 31 03:35:18.710931 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.710685    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-tmp-dir\") pod \"3947643c-111e-4350-9749-47643517df5c\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") "
Oct 31 03:35:18.710931 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.710820    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-dshm\") pod \"3947643c-111e-4350-9749-47643517df5c\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") "
Oct 31 03:35:18.711052 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.710994    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-model-cache" pod "3947643c-111e-4350-9749-47643517df5c" (UID: "3947643c-111e-4350-9749-47643517df5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:35:18.711105 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.711077    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-tmp-dir" pod "3947643c-111e-4350-9749-47643517df5c" (UID: "3947643c-111e-4350-9749-47643517df5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:35:18.711105 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.711092    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-home\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-home\") pod \"3947643c-111e-4350-9749-47643517df5c\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") "
Oct 31 03:35:18.711213 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.711129    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3947643c-111e-4350-9749-47643517df5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/3947643c-111e-4350-9749-47643517df5c-tls-certs\") pod \"3947643c-111e-4350-9749-47643517df5c\" (UID: \"3947643c-111e-4350-9749-47643517df5c\") "
Oct 31 03:35:18.711352 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.711328    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-home" pod "3947643c-111e-4350-9749-47643517df5c" (UID: "3947643c-111e-4350-9749-47643517df5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:35:18.712485 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.712456    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:18.712586 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.712508    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:18.712586 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.712525    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:18.715466 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.714027    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3947643c-111e-4350-9749-47643517df5c-tls-certs" pod "3947643c-111e-4350-9749-47643517df5c" (UID: "3947643c-111e-4350-9749-47643517df5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:35:18.715466 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.714118    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-dshm" pod "3947643c-111e-4350-9749-47643517df5c" (UID: "3947643c-111e-4350-9749-47643517df5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:35:18.715466 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.714246    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3947643c-111e-4350-9749-47643517df5c-kube-api-access-8srnh" pod "3947643c-111e-4350-9749-47643517df5c" (UID: "3947643c-111e-4350-9749-47643517df5c"). InnerVolumeSpecName "kube-api-access-8srnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:35:18.813606 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.813519    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3947643c-111e-4350-9749-47643517df5c-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:18.813606 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.813555    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8srnh\" (UniqueName: \"kubernetes.io/projected/3947643c-111e-4350-9749-47643517df5c-kube-api-access-8srnh\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:18.813606 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.813566    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3947643c-111e-4350-9749-47643517df5c-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:18.881070 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.881035    5415 generic.go:356] "Generic (PLEG): container finished" podID="3947643c-111e-4350-9749-47643517df5c" containerID="e91cb74f9db26b45fdb5981cf6a4289746a7091dc9d9029b008915b25fc65e06" exitCode=0
Oct 31 03:35:18.881257 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.881079    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp" event={"ID":"3947643c-111e-4350-9749-47643517df5c","Type":"ContainerDied","Data":"e91cb74f9db26b45fdb5981cf6a4289746a7091dc9d9029b008915b25fc65e06"}
Oct 31 03:35:18.881257 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.881106    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"
Oct 31 03:35:18.881257 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.881117    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp" event={"ID":"3947643c-111e-4350-9749-47643517df5c","Type":"ContainerDied","Data":"8dbd29a8d2bc34cf115e9c38e641c77c1c6eda8fce66a645751461e3cf8d5457"}
Oct 31 03:35:18.881257 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.881140    5415 scope.go:122] "RemoveContainer" containerID="e91cb74f9db26b45fdb5981cf6a4289746a7091dc9d9029b008915b25fc65e06"
Oct 31 03:35:18.893841 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.893813    5415 scope.go:122] "RemoveContainer" containerID="e91cb74f9db26b45fdb5981cf6a4289746a7091dc9d9029b008915b25fc65e06"
Oct 31 03:35:18.894194 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:35:18.894174    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91cb74f9db26b45fdb5981cf6a4289746a7091dc9d9029b008915b25fc65e06\": container with ID starting with e91cb74f9db26b45fdb5981cf6a4289746a7091dc9d9029b008915b25fc65e06 not found: ID does not exist" containerID="e91cb74f9db26b45fdb5981cf6a4289746a7091dc9d9029b008915b25fc65e06"
Oct 31 03:35:18.894250 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:18.894204    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91cb74f9db26b45fdb5981cf6a4289746a7091dc9d9029b008915b25fc65e06"} err="failed to get container status \"e91cb74f9db26b45fdb5981cf6a4289746a7091dc9d9029b008915b25fc65e06\": rpc error: code = NotFound desc = could not find container \"e91cb74f9db26b45fdb5981cf6a4289746a7091dc9d9029b008915b25fc65e06\": container with ID starting with e91cb74f9db26b45fdb5981cf6a4289746a7091dc9d9029b008915b25fc65e06 not found: ID does not exist"
Oct 31 03:35:19.037814 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:19.037776    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"]
Oct 31 03:35:19.043157 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:19.043126    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp"]
Oct 31 03:35:19.445901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:19.445839    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-6b3a29ec/e2e-lifecycle-model-6b3a29ec-kserve-76dc468f65-6rcnp" podUID="3947643c-111e-4350-9749-47643517df5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:35:20.606367 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:35:20.606331    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3947643c-111e-4350-9749-47643517df5c" path="/var/lib/kubelet/pods/3947643c-111e-4350-9749-47643517df5c/volumes"
Oct 31 03:35:46.176247 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:35:46.176210    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:36:00.237179 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.237137    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"]
Oct 31 03:36:00.237749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.237732    5415 cpu_manager.go:414] "containerMap: removing container" podUID="3947643c-111e-4350-9749-47643517df5c" containerName="main"
Oct 31 03:36:00.237749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.237749    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3947643c-111e-4350-9749-47643517df5c" containerName="main"
Oct 31 03:36:00.237919 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.237899    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="3947643c-111e-4350-9749-47643517df5c" containerName="main"
Oct 31 03:36:00.241220 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.241204    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.290635 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.290596    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e1aad0-openshift-default-dockercfg-tlhkf\""
Oct 31 03:36:00.403592 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.403507    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istiod-ca-cert\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.403592 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.403553    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-podinfo\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.403787 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.403642    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-certs\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.403787 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.403746    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.403787 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.403772    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-data\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.403930 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.403837    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-token\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.403973 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.403943    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxj4s\" (UniqueName: \"kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-kube-api-access-fxj4s\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.404019 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.404006    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-envoy\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.404061 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.404048    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-credential-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.442657 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.442597    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"]
Oct 31 03:36:00.505302 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.505269    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-credential-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.505521 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.505376    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istiod-ca-cert\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.505521 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.505420    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-podinfo\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.505660 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.505571    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-certs\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.505718 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.505700    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.505782 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.505734    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-data\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.505782 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.505769    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-credential-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.505907 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.505791    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-token\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.505976 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.505926    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxj4s\" (UniqueName: \"kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-kube-api-access-fxj4s\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.505976 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.505947    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-certs\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.506087 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.506015    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-envoy\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.506179 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.506120    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.506243 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.506216    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istiod-ca-cert\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.506302 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.506221    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-data\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.508459 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.508432    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-envoy\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.508763 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.508697    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-podinfo\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.562386 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.562355    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-token\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.565585 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.565542    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxj4s\" (UniqueName: \"kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-kube-api-access-fxj4s\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:00.855454 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:00.855422    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:01.280343 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:01.280253    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"]
Oct 31 03:36:01.316195 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:36:01.316162    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fcb6dd5_4b5c_4aaf_83ce_9f4fd63ad4b0.slice/crio-2e1d211e5c0b6dfcf6b60874a79e517f25bf05e5378deb61bf6da858904215a0 WatchSource:0}: Error finding container 2e1d211e5c0b6dfcf6b60874a79e517f25bf05e5378deb61bf6da858904215a0: Status 404 returned error can't find the container with id 2e1d211e5c0b6dfcf6b60874a79e517f25bf05e5378deb61bf6da858904215a0
Oct 31 03:36:01.318636 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:01.318600    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:36:01.318731 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:01.318678    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:36:01.318731 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:01.318708    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:36:02.164332 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:02.164252    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb" event={"ID":"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0","Type":"ContainerStarted","Data":"4f2a80a2c84d1b09e5d8c372a02ee04449c5ebb46e5c6eb781484d66affe746b"}
Oct 31 03:36:02.164819 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:02.164779    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb" event={"ID":"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0","Type":"ContainerStarted","Data":"2e1d211e5c0b6dfcf6b60874a79e517f25bf05e5378deb61bf6da858904215a0"}
Oct 31 03:36:02.327532 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:02.327472    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb" podStartSLOduration=2.327455222 podStartE2EDuration="2.327455222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:02.325454414 +0000 UTC m=+1703.885934250" watchObservedRunningTime="2026-10-31 03:36:02.327455222 +0000 UTC m=+1703.887935020"
Oct 31 03:36:02.867903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:02.857292    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:02.868135 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:02.868046    5415 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:36:02.868135 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:02.868113    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb" podUID="8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 03:36:03.866904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:03.865341    5415 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:36:03.866904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:03.865412    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb" podUID="8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 03:36:04.865903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:04.856341    5415 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:36:04.865903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:04.856415    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb" podUID="8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 03:36:05.337370 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.337334    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"]
Oct 31 03:36:05.337896 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.337684    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp" podUID="8989cbdb-731f-480c-9821-d9532967a613" containerName="maas-api" containerID="cri-o://aeb92b7f832466cdb4f18f37d1eaf4bbb1fc692bfce00d8d97c1e80e30f5c0ec" gracePeriod=30
Oct 31 03:36:05.800386 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.800079    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:36:05.816933 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.816891    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-maas-api-tls\") pod \"8989cbdb-731f-480c-9821-d9532967a613\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") "
Oct 31 03:36:05.817129 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.817029    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-metrics-certs\") pod \"8989cbdb-731f-480c-9821-d9532967a613\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") "
Oct 31 03:36:05.817129 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.817068    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8989cbdb-731f-480c-9821-d9532967a613-kube-api-access-49wpk\" (UniqueName: \"kubernetes.io/projected/8989cbdb-731f-480c-9821-d9532967a613-kube-api-access-49wpk\") pod \"8989cbdb-731f-480c-9821-d9532967a613\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") "
Oct 31 03:36:05.817258 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.817238    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8989cbdb-731f-480c-9821-d9532967a613-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8989cbdb-731f-480c-9821-d9532967a613-openshift-service-ca\") pod \"8989cbdb-731f-480c-9821-d9532967a613\" (UID: \"8989cbdb-731f-480c-9821-d9532967a613\") "
Oct 31 03:36:05.818316 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.818286    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8989cbdb-731f-480c-9821-d9532967a613-openshift-service-ca" pod "8989cbdb-731f-480c-9821-d9532967a613" (UID: "8989cbdb-731f-480c-9821-d9532967a613"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:05.820011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.819984    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-maas-api-tls" pod "8989cbdb-731f-480c-9821-d9532967a613" (UID: "8989cbdb-731f-480c-9821-d9532967a613"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:36:05.820142 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.820018    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8989cbdb-731f-480c-9821-d9532967a613-kube-api-access-49wpk" pod "8989cbdb-731f-480c-9821-d9532967a613" (UID: "8989cbdb-731f-480c-9821-d9532967a613"). InnerVolumeSpecName "kube-api-access-49wpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:05.821197 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.821175    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-metrics-certs" pod "8989cbdb-731f-480c-9821-d9532967a613" (UID: "8989cbdb-731f-480c-9821-d9532967a613"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:36:05.865735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.865180    5415 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:36:05.865735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.865242    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb" podUID="8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 03:36:05.920138 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.920041    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8989cbdb-731f-480c-9821-d9532967a613-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:05.920138 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.920079    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:05.920138 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.920095    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8989cbdb-731f-480c-9821-d9532967a613-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:05.920138 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:05.920113    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49wpk\" (UniqueName: \"kubernetes.io/projected/8989cbdb-731f-480c-9821-d9532967a613-kube-api-access-49wpk\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:06.196115 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:06.196042    5415 generic.go:356] "Generic (PLEG): container finished" podID="8989cbdb-731f-480c-9821-d9532967a613" containerID="aeb92b7f832466cdb4f18f37d1eaf4bbb1fc692bfce00d8d97c1e80e30f5c0ec" exitCode=0
Oct 31 03:36:06.196265 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:06.196198    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp" event={"ID":"8989cbdb-731f-480c-9821-d9532967a613","Type":"ContainerDied","Data":"aeb92b7f832466cdb4f18f37d1eaf4bbb1fc692bfce00d8d97c1e80e30f5c0ec"}
Oct 31 03:36:06.196265 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:06.196236    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp" event={"ID":"8989cbdb-731f-480c-9821-d9532967a613","Type":"ContainerDied","Data":"9eab8868e62593771a83a47735a3ce2cf82ca2f4abdc02628bd9dc6b4f882d58"}
Oct 31 03:36:06.196265 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:06.196257    5415 scope.go:122] "RemoveContainer" containerID="aeb92b7f832466cdb4f18f37d1eaf4bbb1fc692bfce00d8d97c1e80e30f5c0ec"
Oct 31 03:36:06.196455 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:06.196433    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"
Oct 31 03:36:06.215443 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:06.215422    5415 scope.go:122] "RemoveContainer" containerID="aeb92b7f832466cdb4f18f37d1eaf4bbb1fc692bfce00d8d97c1e80e30f5c0ec"
Oct 31 03:36:06.215747 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:36:06.215724    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb92b7f832466cdb4f18f37d1eaf4bbb1fc692bfce00d8d97c1e80e30f5c0ec\": container with ID starting with aeb92b7f832466cdb4f18f37d1eaf4bbb1fc692bfce00d8d97c1e80e30f5c0ec not found: ID does not exist" containerID="aeb92b7f832466cdb4f18f37d1eaf4bbb1fc692bfce00d8d97c1e80e30f5c0ec"
Oct 31 03:36:06.215795 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:06.215757    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb92b7f832466cdb4f18f37d1eaf4bbb1fc692bfce00d8d97c1e80e30f5c0ec"} err="failed to get container status \"aeb92b7f832466cdb4f18f37d1eaf4bbb1fc692bfce00d8d97c1e80e30f5c0ec\": rpc error: code = NotFound desc = could not find container \"aeb92b7f832466cdb4f18f37d1eaf4bbb1fc692bfce00d8d97c1e80e30f5c0ec\": container with ID starting with aeb92b7f832466cdb4f18f37d1eaf4bbb1fc692bfce00d8d97c1e80e30f5c0ec not found: ID does not exist"
Oct 31 03:36:06.374213 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:06.374177    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"]
Oct 31 03:36:06.376831 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:06.376798    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5effde-5f6bf544d6-sqdzp"]
Oct 31 03:36:06.608513 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:06.608481    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8989cbdb-731f-480c-9821-d9532967a613" path="/var/lib/kubelet/pods/8989cbdb-731f-480c-9821-d9532967a613/volumes"
Oct 31 03:36:06.858363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:06.858325    5415 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:36:06.858715 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:06.858649    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb" podUID="8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 03:36:07.861562 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:07.861524    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:07.862045 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:07.861855    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:07.862687 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:07.862665    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:36:16.186187 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:36:16.186100    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:36:29.053807 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.053766    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"]
Oct 31 03:36:29.054564 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.054539    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8989cbdb-731f-480c-9821-d9532967a613" containerName="maas-api"
Oct 31 03:36:29.054663 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.054566    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8989cbdb-731f-480c-9821-d9532967a613" containerName="maas-api"
Oct 31 03:36:29.054740 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.054725    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8989cbdb-731f-480c-9821-d9532967a613" containerName="maas-api"
Oct 31 03:36:29.058222 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.058199    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"
Oct 31 03:36:29.107907 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.107840    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:36:29.155285 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.155231    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:36:29.217999 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.216644    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhpdb\" (UniqueName: \"kubernetes.io/projected/c069a9da-d367-4360-8b3d-4d035364a9e3-kube-api-access-xhpdb\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b\" (UID: \"c069a9da-d367-4360-8b3d-4d035364a9e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"
Oct 31 03:36:29.217999 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.216786    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b\" (UID: \"c069a9da-d367-4360-8b3d-4d035364a9e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"
Oct 31 03:36:29.217999 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.216900    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b\" (UID: \"c069a9da-d367-4360-8b3d-4d035364a9e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"
Oct 31 03:36:29.228349 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.228296    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"]
Oct 31 03:36:29.318260 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.318161    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhpdb\" (UniqueName: \"kubernetes.io/projected/c069a9da-d367-4360-8b3d-4d035364a9e3-kube-api-access-xhpdb\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b\" (UID: \"c069a9da-d367-4360-8b3d-4d035364a9e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"
Oct 31 03:36:29.318440 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.318267    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b\" (UID: \"c069a9da-d367-4360-8b3d-4d035364a9e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"
Oct 31 03:36:29.318440 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.318338    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b\" (UID: \"c069a9da-d367-4360-8b3d-4d035364a9e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"
Oct 31 03:36:29.319149 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.319118    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b\" (UID: \"c069a9da-d367-4360-8b3d-4d035364a9e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"
Oct 31 03:36:29.326779 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.326674    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b\" (UID: \"c069a9da-d367-4360-8b3d-4d035364a9e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"
Oct 31 03:36:29.373483 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.373440    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhpdb\" (UniqueName: \"kubernetes.io/projected/c069a9da-d367-4360-8b3d-4d035364a9e3-kube-api-access-xhpdb\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b\" (UID: \"c069a9da-d367-4360-8b3d-4d035364a9e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"
Oct 31 03:36:29.673077 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:29.673038    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"
Oct 31 03:36:30.045205 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:30.045177    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"]
Oct 31 03:36:30.047429 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:36:30.047396    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc069a9da_d367_4360_8b3d_4d035364a9e3.slice/crio-02eb3ccd215efc4f8cb5b309cdf0a3f852f334319196608e55d797a94da0ed59 WatchSource:0}: Error finding container 02eb3ccd215efc4f8cb5b309cdf0a3f852f334319196608e55d797a94da0ed59: Status 404 returned error can't find the container with id 02eb3ccd215efc4f8cb5b309cdf0a3f852f334319196608e55d797a94da0ed59
Oct 31 03:36:30.350465 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:30.350421    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b" event={"ID":"c069a9da-d367-4360-8b3d-4d035364a9e3","Type":"ContainerStarted","Data":"02eb3ccd215efc4f8cb5b309cdf0a3f852f334319196608e55d797a94da0ed59"}
Oct 31 03:36:31.357286 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:31.357254    5415 generic.go:356] "Generic (PLEG): container finished" podID="c069a9da-d367-4360-8b3d-4d035364a9e3" containerID="ac8b86d5e687e91ac74eae65935238798d615b3cc4d78cdda64e5cd61e8074e0" exitCode=0
Oct 31 03:36:31.357762 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:31.357329    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b" event={"ID":"c069a9da-d367-4360-8b3d-4d035364a9e3","Type":"ContainerDied","Data":"ac8b86d5e687e91ac74eae65935238798d615b3cc4d78cdda64e5cd61e8074e0"}
Oct 31 03:36:32.547677 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:32.547650    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"
Oct 31 03:36:32.560688 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:32.560657    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-xks-ca\") pod \"c069a9da-d367-4360-8b3d-4d035364a9e3\" (UID: \"c069a9da-d367-4360-8b3d-4d035364a9e3\") "
Oct 31 03:36:32.560839 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:32.560747    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-openshift-ca\") pod \"c069a9da-d367-4360-8b3d-4d035364a9e3\" (UID: \"c069a9da-d367-4360-8b3d-4d035364a9e3\") "
Oct 31 03:36:32.560924 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:32.560911    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c069a9da-d367-4360-8b3d-4d035364a9e3-kube-api-access-xhpdb\" (UniqueName: \"kubernetes.io/projected/c069a9da-d367-4360-8b3d-4d035364a9e3-kube-api-access-xhpdb\") pod \"c069a9da-d367-4360-8b3d-4d035364a9e3\" (UID: \"c069a9da-d367-4360-8b3d-4d035364a9e3\") "
Oct 31 03:36:32.561117 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:32.561092    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-openshift-ca" pod "c069a9da-d367-4360-8b3d-4d035364a9e3" (UID: "c069a9da-d367-4360-8b3d-4d035364a9e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:32.561406 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:32.561386    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:32.562975 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:32.562945    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c069a9da-d367-4360-8b3d-4d035364a9e3-kube-api-access-xhpdb" pod "c069a9da-d367-4360-8b3d-4d035364a9e3" (UID: "c069a9da-d367-4360-8b3d-4d035364a9e3"). InnerVolumeSpecName "kube-api-access-xhpdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:32.562975 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:32.562955    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-xks-ca" pod "c069a9da-d367-4360-8b3d-4d035364a9e3" (UID: "c069a9da-d367-4360-8b3d-4d035364a9e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:36:32.663013 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:32.662974    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhpdb\" (UniqueName: \"kubernetes.io/projected/c069a9da-d367-4360-8b3d-4d035364a9e3-kube-api-access-xhpdb\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:32.663013 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:32.663009    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c069a9da-d367-4360-8b3d-4d035364a9e3-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:33.372009 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:33.371979    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"
Oct 31 03:36:33.372225 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:33.372014    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b" event={"ID":"c069a9da-d367-4360-8b3d-4d035364a9e3","Type":"ContainerDied","Data":"02eb3ccd215efc4f8cb5b309cdf0a3f852f334319196608e55d797a94da0ed59"}
Oct 31 03:36:33.372225 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:36:33.372078    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02eb3ccd215efc4f8cb5b309cdf0a3f852f334319196608e55d797a94da0ed59"
Oct 31 03:36:46.176598 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:36:46.176561    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:37:16.176506 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:37:16.176461    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:37:28.436241 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:28.436185    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"]
Oct 31 03:37:28.436842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:28.436451    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt" podUID="515331d3-50e0-4935-a5ae-860f67116f6e" containerName="istio-proxy" containerID="cri-o://0e8754f32c05492bdd76ab4d194955cc44f2901cee8a745c3bc4314978ebaaf4" gracePeriod=30
Oct 31 03:37:31.151916 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:31.151863    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"]
Oct 31 03:37:31.156890 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:31.156847    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tdf5b"]
Oct 31 03:37:31.364664 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:31.364621    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"]
Oct 31 03:37:31.365314 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:31.365252    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt" podUID="db75d372-8a01-458f-95f3-fe20a249d47f" containerName="istio-proxy" containerID="cri-o://7c6cb106c88c8c1e59446bd6381b1aaea59be7316cc7f2adb4f18799de9e944d" gracePeriod=30
Oct 31 03:37:31.368194 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:31.368159    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"]
Oct 31 03:37:31.371053 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:31.370990    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s" podUID="03f3f891-01ae-4bc1-845f-43da70087727" containerName="istio-proxy" containerID="cri-o://eb0919564d62b0cab6f98b2431ee4c7fde6874fdd766b02dec14ab0261ad1312" gracePeriod=30
Oct 31 03:37:32.608810 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:32.608771    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c069a9da-d367-4360-8b3d-4d035364a9e3" path="/var/lib/kubelet/pods/c069a9da-d367-4360-8b3d-4d035364a9e3/volumes"
Oct 31 03:37:33.747266 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.747231    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:37:33.772526 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.772480    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-envoy\") pod \"515331d3-50e0-4935-a5ae-860f67116f6e\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") "
Oct 31 03:37:33.772743 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.772559    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-certs\") pod \"515331d3-50e0-4935-a5ae-860f67116f6e\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") "
Oct 31 03:37:33.772743 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.772597    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/515331d3-50e0-4935-a5ae-860f67116f6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515331d3-50e0-4935-a5ae-860f67116f6e-istiod-ca-cert\") pod \"515331d3-50e0-4935-a5ae-860f67116f6e\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") "
Oct 31 03:37:33.772899 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.772752    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-istio-token\" (UniqueName: \"kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-istio-token\") pod \"515331d3-50e0-4935-a5ae-860f67116f6e\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") "
Oct 31 03:37:33.772899 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.772830    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-credential-socket\") pod \"515331d3-50e0-4935-a5ae-860f67116f6e\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") "
Oct 31 03:37:33.773014 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.772970    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-kube-api-access-qsgr8\" (UniqueName: \"kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-kube-api-access-qsgr8\") pod \"515331d3-50e0-4935-a5ae-860f67116f6e\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") "
Oct 31 03:37:33.773071 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.773039    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-socket\") pod \"515331d3-50e0-4935-a5ae-860f67116f6e\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") "
Oct 31 03:37:33.773127 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.773072    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-data\") pod \"515331d3-50e0-4935-a5ae-860f67116f6e\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") "
Oct 31 03:37:33.773127 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.773113    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/515331d3-50e0-4935-a5ae-860f67116f6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515331d3-50e0-4935-a5ae-860f67116f6e-istio-podinfo\") pod \"515331d3-50e0-4935-a5ae-860f67116f6e\" (UID: \"515331d3-50e0-4935-a5ae-860f67116f6e\") "
Oct 31 03:37:33.773231 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.773146    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-credential-socket" pod "515331d3-50e0-4935-a5ae-860f67116f6e" (UID: "515331d3-50e0-4935-a5ae-860f67116f6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:33.773231 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.773071    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515331d3-50e0-4935-a5ae-860f67116f6e-istiod-ca-cert" pod "515331d3-50e0-4935-a5ae-860f67116f6e" (UID: "515331d3-50e0-4935-a5ae-860f67116f6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:33.773374 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.773352    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-socket" pod "515331d3-50e0-4935-a5ae-860f67116f6e" (UID: "515331d3-50e0-4935-a5ae-860f67116f6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:33.774143 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.772888    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-certs" pod "515331d3-50e0-4935-a5ae-860f67116f6e" (UID: "515331d3-50e0-4935-a5ae-860f67116f6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:33.774143 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.773542    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-data" pod "515331d3-50e0-4935-a5ae-860f67116f6e" (UID: "515331d3-50e0-4935-a5ae-860f67116f6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:33.774143 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.774045    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.774143 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.774070    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515331d3-50e0-4935-a5ae-860f67116f6e-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.774143 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.774088    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.774143 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.774103    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.774143 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.774117    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.775552 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.775520    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-istio-token" pod "515331d3-50e0-4935-a5ae-860f67116f6e" (UID: "515331d3-50e0-4935-a5ae-860f67116f6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:33.775687 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.775635    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/515331d3-50e0-4935-a5ae-860f67116f6e-istio-podinfo" pod "515331d3-50e0-4935-a5ae-860f67116f6e" (UID: "515331d3-50e0-4935-a5ae-860f67116f6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:33.775855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.775834    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-envoy" pod "515331d3-50e0-4935-a5ae-860f67116f6e" (UID: "515331d3-50e0-4935-a5ae-860f67116f6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:33.775945 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.775855    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-kube-api-access-qsgr8" pod "515331d3-50e0-4935-a5ae-860f67116f6e" (UID: "515331d3-50e0-4935-a5ae-860f67116f6e"). InnerVolumeSpecName "kube-api-access-qsgr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:33.875527 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.875495    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515331d3-50e0-4935-a5ae-860f67116f6e-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.875527 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.875522    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515331d3-50e0-4935-a5ae-860f67116f6e-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.875527 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.875534    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.875820 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.875543    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsgr8\" (UniqueName: \"kubernetes.io/projected/515331d3-50e0-4935-a5ae-860f67116f6e-kube-api-access-qsgr8\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.876508 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.876477    5415 generic.go:356] "Generic (PLEG): container finished" podID="515331d3-50e0-4935-a5ae-860f67116f6e" containerID="0e8754f32c05492bdd76ab4d194955cc44f2901cee8a745c3bc4314978ebaaf4" exitCode=0
Oct 31 03:37:33.876633 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.876529    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"
Oct 31 03:37:33.876633 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.876567    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt" event={"ID":"515331d3-50e0-4935-a5ae-860f67116f6e","Type":"ContainerDied","Data":"0e8754f32c05492bdd76ab4d194955cc44f2901cee8a745c3bc4314978ebaaf4"}
Oct 31 03:37:33.876706 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.876639    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt" event={"ID":"515331d3-50e0-4935-a5ae-860f67116f6e","Type":"ContainerDied","Data":"764b89163c724e7fb542c6e329cca14a4ff0df8a8696a337d9b929340c254d7f"}
Oct 31 03:37:33.876706 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.876674    5415 scope.go:122] "RemoveContainer" containerID="0e8754f32c05492bdd76ab4d194955cc44f2901cee8a745c3bc4314978ebaaf4"
Oct 31 03:37:33.888022 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.888002    5415 scope.go:122] "RemoveContainer" containerID="0e8754f32c05492bdd76ab4d194955cc44f2901cee8a745c3bc4314978ebaaf4"
Oct 31 03:37:33.888361 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:37:33.888336    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e8754f32c05492bdd76ab4d194955cc44f2901cee8a745c3bc4314978ebaaf4\": container with ID starting with 0e8754f32c05492bdd76ab4d194955cc44f2901cee8a745c3bc4314978ebaaf4 not found: ID does not exist" containerID="0e8754f32c05492bdd76ab4d194955cc44f2901cee8a745c3bc4314978ebaaf4"
Oct 31 03:37:33.888434 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:33.888378    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e8754f32c05492bdd76ab4d194955cc44f2901cee8a745c3bc4314978ebaaf4"} err="failed to get container status \"0e8754f32c05492bdd76ab4d194955cc44f2901cee8a745c3bc4314978ebaaf4\": rpc error: code = NotFound desc = could not find container \"0e8754f32c05492bdd76ab4d194955cc44f2901cee8a745c3bc4314978ebaaf4\": container with ID starting with 0e8754f32c05492bdd76ab4d194955cc44f2901cee8a745c3bc4314978ebaaf4 not found: ID does not exist"
Oct 31 03:37:34.013855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:34.012450    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"]
Oct 31 03:37:34.019494 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:34.019451    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-scjzt"]
Oct 31 03:37:34.606478 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:34.606440    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="515331d3-50e0-4935-a5ae-860f67116f6e" path="/var/lib/kubelet/pods/515331d3-50e0-4935-a5ae-860f67116f6e/volumes"
Oct 31 03:37:36.674233 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.674196    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:37:36.712197 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.712164    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-credential-socket\") pod \"db75d372-8a01-458f-95f3-fe20a249d47f\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") "
Oct 31 03:37:36.712381 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.712242    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db75d372-8a01-458f-95f3-fe20a249d47f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db75d372-8a01-458f-95f3-fe20a249d47f-istiod-ca-cert\") pod \"db75d372-8a01-458f-95f3-fe20a249d47f\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") "
Oct 31 03:37:36.712381 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.712287    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-kube-api-access-jjck5\" (UniqueName: \"kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-kube-api-access-jjck5\") pod \"db75d372-8a01-458f-95f3-fe20a249d47f\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") "
Oct 31 03:37:36.712381 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.712342    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db75d372-8a01-458f-95f3-fe20a249d47f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db75d372-8a01-458f-95f3-fe20a249d47f-istio-podinfo\") pod \"db75d372-8a01-458f-95f3-fe20a249d47f\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") "
Oct 31 03:37:36.712576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.712468    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-socket\") pod \"db75d372-8a01-458f-95f3-fe20a249d47f\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") "
Oct 31 03:37:36.712576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.712541    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-data\") pod \"db75d372-8a01-458f-95f3-fe20a249d47f\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") "
Oct 31 03:37:36.712576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.712521    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-credential-socket" pod "db75d372-8a01-458f-95f3-fe20a249d47f" (UID: "db75d372-8a01-458f-95f3-fe20a249d47f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.712733 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.712586    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-envoy\") pod \"db75d372-8a01-458f-95f3-fe20a249d47f\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") "
Oct 31 03:37:36.712733 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.712663    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-certs\") pod \"db75d372-8a01-458f-95f3-fe20a249d47f\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") "
Oct 31 03:37:36.712733 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.712711    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-istio-token\" (UniqueName: \"kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-istio-token\") pod \"db75d372-8a01-458f-95f3-fe20a249d47f\" (UID: \"db75d372-8a01-458f-95f3-fe20a249d47f\") "
Oct 31 03:37:36.712907 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.712831    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-data" pod "db75d372-8a01-458f-95f3-fe20a249d47f" (UID: "db75d372-8a01-458f-95f3-fe20a249d47f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.713068 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.713008    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-certs" pod "db75d372-8a01-458f-95f3-fe20a249d47f" (UID: "db75d372-8a01-458f-95f3-fe20a249d47f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.713068 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.713024    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-socket" pod "db75d372-8a01-458f-95f3-fe20a249d47f" (UID: "db75d372-8a01-458f-95f3-fe20a249d47f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.713542 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.713317    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db75d372-8a01-458f-95f3-fe20a249d47f-istiod-ca-cert" pod "db75d372-8a01-458f-95f3-fe20a249d47f" (UID: "db75d372-8a01-458f-95f3-fe20a249d47f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:36.713737 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.713683    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.713737 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.713710    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.713737 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.713725    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.713990 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.713739    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.713990 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.713753    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db75d372-8a01-458f-95f3-fe20a249d47f-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.715039 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.715004    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-istio-token" pod "db75d372-8a01-458f-95f3-fe20a249d47f" (UID: "db75d372-8a01-458f-95f3-fe20a249d47f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:36.715271 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.715247    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db75d372-8a01-458f-95f3-fe20a249d47f-istio-podinfo" pod "db75d372-8a01-458f-95f3-fe20a249d47f" (UID: "db75d372-8a01-458f-95f3-fe20a249d47f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:36.715271 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.715262    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-kube-api-access-jjck5" pod "db75d372-8a01-458f-95f3-fe20a249d47f" (UID: "db75d372-8a01-458f-95f3-fe20a249d47f"). InnerVolumeSpecName "kube-api-access-jjck5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:36.715624 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.715603    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-envoy" pod "db75d372-8a01-458f-95f3-fe20a249d47f" (UID: "db75d372-8a01-458f-95f3-fe20a249d47f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.756736 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.756706    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:37:36.814217 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.814177    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-istio-token\" (UniqueName: \"kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-istio-token\") pod \"03f3f891-01ae-4bc1-845f-43da70087727\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") "
Oct 31 03:37:36.814410 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.814233    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03f3f891-01ae-4bc1-845f-43da70087727-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f3f891-01ae-4bc1-845f-43da70087727-istio-podinfo\") pod \"03f3f891-01ae-4bc1-845f-43da70087727\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") "
Oct 31 03:37:36.814410 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.814273    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-socket\") pod \"03f3f891-01ae-4bc1-845f-43da70087727\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") "
Oct 31 03:37:36.814410 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.814374    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f3f891-01ae-4bc1-845f-43da70087727-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f3f891-01ae-4bc1-845f-43da70087727-istiod-ca-cert\") pod \"03f3f891-01ae-4bc1-845f-43da70087727\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") "
Oct 31 03:37:36.814537 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.814485    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-data\") pod \"03f3f891-01ae-4bc1-845f-43da70087727\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") "
Oct 31 03:37:36.814537 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.814521    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-credential-socket\") pod \"03f3f891-01ae-4bc1-845f-43da70087727\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") "
Oct 31 03:37:36.814618 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.814594    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-socket" pod "03f3f891-01ae-4bc1-845f-43da70087727" (UID: "03f3f891-01ae-4bc1-845f-43da70087727"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.814674 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.814622    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-envoy\") pod \"03f3f891-01ae-4bc1-845f-43da70087727\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") "
Oct 31 03:37:36.814736 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.814676    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-kube-api-access-k7tq2\" (UniqueName: \"kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-kube-api-access-k7tq2\") pod \"03f3f891-01ae-4bc1-845f-43da70087727\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") "
Oct 31 03:37:36.814736 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.814707    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-certs\") pod \"03f3f891-01ae-4bc1-845f-43da70087727\" (UID: \"03f3f891-01ae-4bc1-845f-43da70087727\") "
Oct 31 03:37:36.814837 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.814763    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-data" pod "03f3f891-01ae-4bc1-845f-43da70087727" (UID: "03f3f891-01ae-4bc1-845f-43da70087727"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.814837 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.814809    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f3f891-01ae-4bc1-845f-43da70087727-istiod-ca-cert" pod "03f3f891-01ae-4bc1-845f-43da70087727" (UID: "03f3f891-01ae-4bc1-845f-43da70087727"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:36.814991 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.814837    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-credential-socket" pod "03f3f891-01ae-4bc1-845f-43da70087727" (UID: "03f3f891-01ae-4bc1-845f-43da70087727"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.815113 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.815087    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-certs" pod "03f3f891-01ae-4bc1-845f-43da70087727" (UID: "03f3f891-01ae-4bc1-845f-43da70087727"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.815359 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.815336    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.815468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.815361    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.815468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.815377    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f3f891-01ae-4bc1-845f-43da70087727-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.815468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.815388    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjck5\" (UniqueName: \"kubernetes.io/projected/db75d372-8a01-458f-95f3-fe20a249d47f-kube-api-access-jjck5\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.815468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.815398    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.815468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.815410    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.815468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.815419    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db75d372-8a01-458f-95f3-fe20a249d47f-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.815468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.815429    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.815468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.815438    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db75d372-8a01-458f-95f3-fe20a249d47f-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.816789 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.816770    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-envoy" pod "03f3f891-01ae-4bc1-845f-43da70087727" (UID: "03f3f891-01ae-4bc1-845f-43da70087727"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.816924 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.816904    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-kube-api-access-k7tq2" pod "03f3f891-01ae-4bc1-845f-43da70087727" (UID: "03f3f891-01ae-4bc1-845f-43da70087727"). InnerVolumeSpecName "kube-api-access-k7tq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:36.817018 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.817002    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-istio-token" pod "03f3f891-01ae-4bc1-845f-43da70087727" (UID: "03f3f891-01ae-4bc1-845f-43da70087727"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:36.817068 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.817047    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03f3f891-01ae-4bc1-845f-43da70087727-istio-podinfo" pod "03f3f891-01ae-4bc1-845f-43da70087727" (UID: "03f3f891-01ae-4bc1-845f-43da70087727"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:36.901093 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.900979    5415 generic.go:356] "Generic (PLEG): container finished" podID="db75d372-8a01-458f-95f3-fe20a249d47f" containerID="7c6cb106c88c8c1e59446bd6381b1aaea59be7316cc7f2adb4f18799de9e944d" exitCode=0
Oct 31 03:37:36.901093 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.901041    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"
Oct 31 03:37:36.901323 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.901094    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt" event={"ID":"db75d372-8a01-458f-95f3-fe20a249d47f","Type":"ContainerDied","Data":"7c6cb106c88c8c1e59446bd6381b1aaea59be7316cc7f2adb4f18799de9e944d"}
Oct 31 03:37:36.901323 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.901174    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt" event={"ID":"db75d372-8a01-458f-95f3-fe20a249d47f","Type":"ContainerDied","Data":"39ed4b533d0bf65b80cf978fe5a904ae255b4921ca1f03e4b9bdd4e4e6d5aa44"}
Oct 31 03:37:36.901323 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.901201    5415 scope.go:122] "RemoveContainer" containerID="7c6cb106c88c8c1e59446bd6381b1aaea59be7316cc7f2adb4f18799de9e944d"
Oct 31 03:37:36.903752 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.903722    5415 generic.go:356] "Generic (PLEG): container finished" podID="03f3f891-01ae-4bc1-845f-43da70087727" containerID="eb0919564d62b0cab6f98b2431ee4c7fde6874fdd766b02dec14ab0261ad1312" exitCode=0
Oct 31 03:37:36.903950 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.903772    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"
Oct 31 03:37:36.903950 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.903858    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s" event={"ID":"03f3f891-01ae-4bc1-845f-43da70087727","Type":"ContainerDied","Data":"eb0919564d62b0cab6f98b2431ee4c7fde6874fdd766b02dec14ab0261ad1312"}
Oct 31 03:37:36.904069 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.903946    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s" event={"ID":"03f3f891-01ae-4bc1-845f-43da70087727","Type":"ContainerDied","Data":"bb70e332346ccd1243910670ed1fb6e5b5bab88a73f189f169cd1a19b3228acb"}
Oct 31 03:37:36.915861 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.915835    5415 scope.go:122] "RemoveContainer" containerID="7c6cb106c88c8c1e59446bd6381b1aaea59be7316cc7f2adb4f18799de9e944d"
Oct 31 03:37:36.916576 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:37:36.916278    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c6cb106c88c8c1e59446bd6381b1aaea59be7316cc7f2adb4f18799de9e944d\": container with ID starting with 7c6cb106c88c8c1e59446bd6381b1aaea59be7316cc7f2adb4f18799de9e944d not found: ID does not exist" containerID="7c6cb106c88c8c1e59446bd6381b1aaea59be7316cc7f2adb4f18799de9e944d"
Oct 31 03:37:36.916576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.916325    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6cb106c88c8c1e59446bd6381b1aaea59be7316cc7f2adb4f18799de9e944d"} err="failed to get container status \"7c6cb106c88c8c1e59446bd6381b1aaea59be7316cc7f2adb4f18799de9e944d\": rpc error: code = NotFound desc = could not find container \"7c6cb106c88c8c1e59446bd6381b1aaea59be7316cc7f2adb4f18799de9e944d\": container with ID starting with 7c6cb106c88c8c1e59446bd6381b1aaea59be7316cc7f2adb4f18799de9e944d not found: ID does not exist"
Oct 31 03:37:36.916576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.916351    5415 scope.go:122] "RemoveContainer" containerID="eb0919564d62b0cab6f98b2431ee4c7fde6874fdd766b02dec14ab0261ad1312"
Oct 31 03:37:36.917047 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.916923    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.917047 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.916950    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f3f891-01ae-4bc1-845f-43da70087727-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.917047 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.916965    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f3f891-01ae-4bc1-845f-43da70087727-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.917047 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.916981    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7tq2\" (UniqueName: \"kubernetes.io/projected/03f3f891-01ae-4bc1-845f-43da70087727-kube-api-access-k7tq2\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.931169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.930756    5415 scope.go:122] "RemoveContainer" containerID="eb0919564d62b0cab6f98b2431ee4c7fde6874fdd766b02dec14ab0261ad1312"
Oct 31 03:37:36.932602 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:37:36.932479    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0919564d62b0cab6f98b2431ee4c7fde6874fdd766b02dec14ab0261ad1312\": container with ID starting with eb0919564d62b0cab6f98b2431ee4c7fde6874fdd766b02dec14ab0261ad1312 not found: ID does not exist" containerID="eb0919564d62b0cab6f98b2431ee4c7fde6874fdd766b02dec14ab0261ad1312"
Oct 31 03:37:36.932602 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:36.932529    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0919564d62b0cab6f98b2431ee4c7fde6874fdd766b02dec14ab0261ad1312"} err="failed to get container status \"eb0919564d62b0cab6f98b2431ee4c7fde6874fdd766b02dec14ab0261ad1312\": rpc error: code = NotFound desc = could not find container \"eb0919564d62b0cab6f98b2431ee4c7fde6874fdd766b02dec14ab0261ad1312\": container with ID starting with eb0919564d62b0cab6f98b2431ee4c7fde6874fdd766b02dec14ab0261ad1312 not found: ID does not exist"
Oct 31 03:37:37.056505 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:37.056466    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"]
Oct 31 03:37:37.059333 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:37.059297    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-45k6s"]
Oct 31 03:37:37.164592 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:37.164495    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"]
Oct 31 03:37:37.168448 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:37.168403    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bw2lt"]
Oct 31 03:37:38.607409 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:38.607373    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f3f891-01ae-4bc1-845f-43da70087727" path="/var/lib/kubelet/pods/03f3f891-01ae-4bc1-845f-43da70087727/volumes"
Oct 31 03:37:38.607937 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:38.607900    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db75d372-8a01-458f-95f3-fe20a249d47f" path="/var/lib/kubelet/pods/db75d372-8a01-458f-95f3-fe20a249d47f/volumes"
Oct 31 03:37:41.810270 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.810230    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"]
Oct 31 03:37:41.810785 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.810769    5415 cpu_manager.go:414] "containerMap: removing container" podUID="03f3f891-01ae-4bc1-845f-43da70087727" containerName="istio-proxy"
Oct 31 03:37:41.810785 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.810783    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f3f891-01ae-4bc1-845f-43da70087727" containerName="istio-proxy"
Oct 31 03:37:41.810870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.810799    5415 cpu_manager.go:414] "containerMap: removing container" podUID="db75d372-8a01-458f-95f3-fe20a249d47f" containerName="istio-proxy"
Oct 31 03:37:41.810870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.810804    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db75d372-8a01-458f-95f3-fe20a249d47f" containerName="istio-proxy"
Oct 31 03:37:41.810870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.810812    5415 cpu_manager.go:414] "containerMap: removing container" podUID="515331d3-50e0-4935-a5ae-860f67116f6e" containerName="istio-proxy"
Oct 31 03:37:41.810870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.810817    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="515331d3-50e0-4935-a5ae-860f67116f6e" containerName="istio-proxy"
Oct 31 03:37:41.810870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.810839    5415 cpu_manager.go:414] "containerMap: removing container" podUID="c069a9da-d367-4360-8b3d-4d035364a9e3" containerName="revoke-keys"
Oct 31 03:37:41.810870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.810844    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c069a9da-d367-4360-8b3d-4d035364a9e3" containerName="revoke-keys"
Oct 31 03:37:41.811099 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.810940    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="db75d372-8a01-458f-95f3-fe20a249d47f" containerName="istio-proxy"
Oct 31 03:37:41.811099 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.810950    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="515331d3-50e0-4935-a5ae-860f67116f6e" containerName="istio-proxy"
Oct 31 03:37:41.811099 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.810960    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="c069a9da-d367-4360-8b3d-4d035364a9e3" containerName="revoke-keys"
Oct 31 03:37:41.811099 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.810969    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f3f891-01ae-4bc1-845f-43da70087727" containerName="istio-proxy"
Oct 31 03:37:41.816353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.816328    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"
Oct 31 03:37:41.893042 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.891961    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7cef69-f9b2-407d-a283-4f292a19b91b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w\" (UID: \"1b7cef69-f9b2-407d-a283-4f292a19b91b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"
Oct 31 03:37:41.893042 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.892037    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfhcc\" (UniqueName: \"kubernetes.io/projected/1b7cef69-f9b2-407d-a283-4f292a19b91b-kube-api-access-zfhcc\") pod \"payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w\" (UID: \"1b7cef69-f9b2-407d-a283-4f292a19b91b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"
Oct 31 03:37:41.915865 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.915134    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-329d79\""
Oct 31 03:37:41.916084 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.916046    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-329d79-dockercfg-fjzkj\""
Oct 31 03:37:41.977921 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:41.966249    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"]
Oct 31 03:37:42.006903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:42.005677    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfhcc\" (UniqueName: \"kubernetes.io/projected/1b7cef69-f9b2-407d-a283-4f292a19b91b-kube-api-access-zfhcc\") pod \"payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w\" (UID: \"1b7cef69-f9b2-407d-a283-4f292a19b91b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"
Oct 31 03:37:42.006903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:42.005983    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7cef69-f9b2-407d-a283-4f292a19b91b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w\" (UID: \"1b7cef69-f9b2-407d-a283-4f292a19b91b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"
Oct 31 03:37:42.029806 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:42.028262    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7cef69-f9b2-407d-a283-4f292a19b91b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w\" (UID: \"1b7cef69-f9b2-407d-a283-4f292a19b91b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"
Oct 31 03:37:42.065242 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:42.065151    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfhcc\" (UniqueName: \"kubernetes.io/projected/1b7cef69-f9b2-407d-a283-4f292a19b91b-kube-api-access-zfhcc\") pod \"payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w\" (UID: \"1b7cef69-f9b2-407d-a283-4f292a19b91b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"
Oct 31 03:37:42.130972 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:42.130937    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"
Oct 31 03:37:42.503842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:42.503810    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"]
Oct 31 03:37:42.506176 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:37:42.506143    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7cef69_f9b2_407d_a283_4f292a19b91b.slice/crio-837e7a65bed46d90b631386ac665168efff0a3ec42825fba8060bca4cc026246 WatchSource:0}: Error finding container 837e7a65bed46d90b631386ac665168efff0a3ec42825fba8060bca4cc026246: Status 404 returned error can't find the container with id 837e7a65bed46d90b631386ac665168efff0a3ec42825fba8060bca4cc026246
Oct 31 03:37:43.040543 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:43.040497    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w" event={"ID":"1b7cef69-f9b2-407d-a283-4f292a19b91b","Type":"ContainerStarted","Data":"bc0569332ef44d88b31010ffc639ec19d87a1a3d8a6a873c826e764561ddd360"}
Oct 31 03:37:43.041127 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:43.040726    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"
Oct 31 03:37:43.041127 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:43.040789    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w" event={"ID":"1b7cef69-f9b2-407d-a283-4f292a19b91b","Type":"ContainerStarted","Data":"837e7a65bed46d90b631386ac665168efff0a3ec42825fba8060bca4cc026246"}
Oct 31 03:37:43.149228 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:43.149174    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w" podStartSLOduration=2.149158347 podStartE2EDuration="2.149158347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:43.147724506 +0000 UTC m=+1804.708204266" watchObservedRunningTime="2026-10-31 03:37:43.149158347 +0000 UTC m=+1804.709638056"
Oct 31 03:37:46.178035 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:37:46.177954    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:37:47.153435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.153397    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"]
Oct 31 03:37:47.157770 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.157745    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:47.207067 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.207034    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5d3964-serving-cert\""
Oct 31 03:37:47.257421 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.257383    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"]
Oct 31 03:37:47.291607 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.291566    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh2pz\" (UniqueName: \"kubernetes.io/projected/3da717ff-7f08-4393-849b-052cca9e3680-kube-api-access-vh2pz\") pod \"maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:47.291821 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.291746    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:47.291925 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.291839    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da717ff-7f08-4393-849b-052cca9e3680-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:47.291925 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.291869    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-metrics-certs\") pod \"maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:47.393489 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.393429    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:47.393725 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.393508    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da717ff-7f08-4393-849b-052cca9e3680-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:47.393725 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.393537    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-metrics-certs\") pod \"maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:47.393725 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:37:47.393592    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-serving-cert: secret "maas-api-e2e-ipp-b-5d3964-serving-cert" not found
Oct 31 03:37:47.393725 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:37:47.393666    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-maas-api-tls podName:3da717ff-7f08-4393-849b-052cca9e3680 nodeName:}" failed. No retries permitted until 2026-10-31 03:37:47.893648799 +0000 UTC m=+1809.454128507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-maas-api-tls") pod "maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx" (UID: "3da717ff-7f08-4393-849b-052cca9e3680") : secret "maas-api-e2e-ipp-b-5d3964-serving-cert" not found
Oct 31 03:37:47.393725 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.393703    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh2pz\" (UniqueName: \"kubernetes.io/projected/3da717ff-7f08-4393-849b-052cca9e3680-kube-api-access-vh2pz\") pod \"maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:47.394257 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.394239    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da717ff-7f08-4393-849b-052cca9e3680-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:47.396993 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.396959    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-metrics-certs\") pod \"maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:47.453796 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.453703    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh2pz\" (UniqueName: \"kubernetes.io/projected/3da717ff-7f08-4393-849b-052cca9e3680-kube-api-access-vh2pz\") pod \"maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:47.901699 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.901661    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:47.904272 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:47.904236    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:48.072305 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:48.072257    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:48.444685 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:48.444657    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"]
Oct 31 03:37:48.446638 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:37:48.446603    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da717ff_7f08_4393_849b_052cca9e3680.slice/crio-e956abf51a35eb703038d9895dee66a432b4582d32ca8be0f1715f8ff249d729 WatchSource:0}: Error finding container e956abf51a35eb703038d9895dee66a432b4582d32ca8be0f1715f8ff249d729: Status 404 returned error can't find the container with id e956abf51a35eb703038d9895dee66a432b4582d32ca8be0f1715f8ff249d729
Oct 31 03:37:48.448333 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:48.448287    5415 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:37:49.077463 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:49.077427    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx" event={"ID":"3da717ff-7f08-4393-849b-052cca9e3680","Type":"ContainerStarted","Data":"e956abf51a35eb703038d9895dee66a432b4582d32ca8be0f1715f8ff249d729"}
Oct 31 03:37:50.084610 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:50.084573    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx" event={"ID":"3da717ff-7f08-4393-849b-052cca9e3680","Type":"ContainerStarted","Data":"f132450985282b77e355c0d46cae2425b7aa6ab9a511c62013a00b3004cd0253"}
Oct 31 03:37:50.085026 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:50.084746    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:37:50.195355 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:50.195299    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx" podStartSLOduration=2.301849275 podStartE2EDuration="3.195284118s" totalImagesPullingTime="893.434843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:50.192938854 +0000 UTC m=+1811.753418614" watchObservedRunningTime="2026-10-31 03:37:50.195284118 +0000 UTC m=+1811.755763848"
Oct 31 03:37:52.800796 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.800760    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"]
Oct 31 03:37:52.805168 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.805143    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:52.854537 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.854496    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a32cdb-serving-cert\""
Oct 31 03:37:52.865482 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.865435    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p57fv\" (UniqueName: \"kubernetes.io/projected/b5272b1e-7cdc-4d44-a658-50f259988c1e-kube-api-access-p57fv\") pod \"maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:52.865685 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.865559    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-metrics-certs\") pod \"maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:52.865685 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.865617    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:52.865810 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.865731    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5272b1e-7cdc-4d44-a658-50f259988c1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:52.894871 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.894826    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"]
Oct 31 03:37:52.967006 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.966963    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p57fv\" (UniqueName: \"kubernetes.io/projected/b5272b1e-7cdc-4d44-a658-50f259988c1e-kube-api-access-p57fv\") pod \"maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:52.967204 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.967051    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-metrics-certs\") pod \"maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:52.967204 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.967111    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:52.967204 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.967159    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5272b1e-7cdc-4d44-a658-50f259988c1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:52.967522 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:37:52.967502    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-serving-cert: secret "maas-api-e2e-ipp-a-a32cdb-serving-cert" not found
Oct 31 03:37:52.967606 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:37:52.967594    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-maas-api-tls podName:b5272b1e-7cdc-4d44-a658-50f259988c1e nodeName:}" failed. No retries permitted until 2026-10-31 03:37:53.467570261 +0000 UTC m=+1815.028049984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-maas-api-tls") pod "maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955" (UID: "b5272b1e-7cdc-4d44-a658-50f259988c1e") : secret "maas-api-e2e-ipp-a-a32cdb-serving-cert" not found
Oct 31 03:37:52.968256 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.968226    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5272b1e-7cdc-4d44-a658-50f259988c1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:52.970915 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:52.970791    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-metrics-certs\") pod \"maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:53.024122 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:53.024088    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p57fv\" (UniqueName: \"kubernetes.io/projected/b5272b1e-7cdc-4d44-a658-50f259988c1e-kube-api-access-p57fv\") pod \"maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:53.474612 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:53.474572    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:53.477234 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:53.477198    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:53.719305 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:53.719263    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:54.049772 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:54.049686    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"
Oct 31 03:37:54.092914 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:54.092863    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"]
Oct 31 03:37:54.095363 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:37:54.095329    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5272b1e_7cdc_4d44_a658_50f259988c1e.slice/crio-cde885168fcc1aafb7bd0770d6e92a6b09e0bb302494da42acd5d8664411e2a9 WatchSource:0}: Error finding container cde885168fcc1aafb7bd0770d6e92a6b09e0bb302494da42acd5d8664411e2a9: Status 404 returned error can't find the container with id cde885168fcc1aafb7bd0770d6e92a6b09e0bb302494da42acd5d8664411e2a9
Oct 31 03:37:54.112437 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:54.112399    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955" event={"ID":"b5272b1e-7cdc-4d44-a658-50f259988c1e","Type":"ContainerStarted","Data":"cde885168fcc1aafb7bd0770d6e92a6b09e0bb302494da42acd5d8664411e2a9"}
Oct 31 03:37:55.127374 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:55.127328    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955" event={"ID":"b5272b1e-7cdc-4d44-a658-50f259988c1e","Type":"ContainerStarted","Data":"bb7255c5d8676971d0e04577e6b9dc3760efb94cf4390fc775f0dbfc7ae23c40"}
Oct 31 03:37:55.127901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:55.127625    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:37:55.249809 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:55.249755    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955" podStartSLOduration=2.325948751 podStartE2EDuration="3.249738404s" totalImagesPullingTime="923.789653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:55.2476633 +0000 UTC m=+1816.808143065" watchObservedRunningTime="2026-10-31 03:37:55.249738404 +0000 UTC m=+1816.810218155"
Oct 31 03:37:56.098372 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:37:56.098341    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:38:01.140319 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:01.140288    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:38:09.107692 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:09.107658    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"]
Oct 31 03:38:09.108290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:09.107943    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn" podUID="8ae6e719-0fac-495f-b792-aecd2e6fd7cc" containerName="istio-proxy" containerID="cri-o://7754c8fe9636c45be48540d220378291dd092555078b713c2c7e41cef35263ba" gracePeriod=30
Oct 31 03:38:14.261904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.261856    5415 generic.go:356] "Generic (PLEG): container finished" podID="8ae6e719-0fac-495f-b792-aecd2e6fd7cc" containerID="7754c8fe9636c45be48540d220378291dd092555078b713c2c7e41cef35263ba" exitCode=0
Oct 31 03:38:14.262299 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.261913    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn" event={"ID":"8ae6e719-0fac-495f-b792-aecd2e6fd7cc","Type":"ContainerDied","Data":"7754c8fe9636c45be48540d220378291dd092555078b713c2c7e41cef35263ba"}
Oct 31 03:38:14.385316 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.385243    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:38:14.450968 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.450936    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-kube-api-access-glz67\" (UniqueName: \"kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-kube-api-access-glz67\") pod \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") "
Oct 31 03:38:14.451166 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.451032    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-token\" (UniqueName: \"kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-token\") pod \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") "
Oct 31 03:38:14.451166 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.451087    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istiod-ca-cert\") pod \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") "
Oct 31 03:38:14.451166 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.451133    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-certs\") pod \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") "
Oct 31 03:38:14.451344 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.451186    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-credential-socket\") pod \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") "
Oct 31 03:38:14.451344 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.451244    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-socket\") pod \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") "
Oct 31 03:38:14.451344 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.451302    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-podinfo\") pod \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") "
Oct 31 03:38:14.451500 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.451480    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-certs" pod "8ae6e719-0fac-495f-b792-aecd2e6fd7cc" (UID: "8ae6e719-0fac-495f-b792-aecd2e6fd7cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:14.451571 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.451490    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-credential-socket" pod "8ae6e719-0fac-495f-b792-aecd2e6fd7cc" (UID: "8ae6e719-0fac-495f-b792-aecd2e6fd7cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:14.451624 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.451559    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-socket" pod "8ae6e719-0fac-495f-b792-aecd2e6fd7cc" (UID: "8ae6e719-0fac-495f-b792-aecd2e6fd7cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:14.451624 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.451585    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istiod-ca-cert" pod "8ae6e719-0fac-495f-b792-aecd2e6fd7cc" (UID: "8ae6e719-0fac-495f-b792-aecd2e6fd7cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:14.451759 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.451736    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-data\") pod \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") "
Oct 31 03:38:14.451972 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.451768    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-envoy\") pod \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\" (UID: \"8ae6e719-0fac-495f-b792-aecd2e6fd7cc\") "
Oct 31 03:38:14.452047 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.452011    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-data" pod "8ae6e719-0fac-495f-b792-aecd2e6fd7cc" (UID: "8ae6e719-0fac-495f-b792-aecd2e6fd7cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:14.452302 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.452284    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:14.452386 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.452302    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:14.452386 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.452315    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:14.452386 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.452327    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:14.452386 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.452337    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:14.453576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.453552    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-podinfo" pod "8ae6e719-0fac-495f-b792-aecd2e6fd7cc" (UID: "8ae6e719-0fac-495f-b792-aecd2e6fd7cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:38:14.453870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.453844    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-token" pod "8ae6e719-0fac-495f-b792-aecd2e6fd7cc" (UID: "8ae6e719-0fac-495f-b792-aecd2e6fd7cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:14.453975 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.453854    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-kube-api-access-glz67" pod "8ae6e719-0fac-495f-b792-aecd2e6fd7cc" (UID: "8ae6e719-0fac-495f-b792-aecd2e6fd7cc"). InnerVolumeSpecName "kube-api-access-glz67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:14.453975 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.453916    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-envoy" pod "8ae6e719-0fac-495f-b792-aecd2e6fd7cc" (UID: "8ae6e719-0fac-495f-b792-aecd2e6fd7cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:14.554149 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.554111    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:14.554149 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.554142    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:14.554149 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.554153    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:14.554408 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:14.554163    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glz67\" (UniqueName: \"kubernetes.io/projected/8ae6e719-0fac-495f-b792-aecd2e6fd7cc-kube-api-access-glz67\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:15.270764 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:15.270735    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"
Oct 31 03:38:15.271292 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:15.270757    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn" event={"ID":"8ae6e719-0fac-495f-b792-aecd2e6fd7cc","Type":"ContainerDied","Data":"a84fccc060dfb35a67f15af6c27bef141214ca3391457d03f79761e489d8e522"}
Oct 31 03:38:15.271292 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:15.270841    5415 scope.go:122] "RemoveContainer" containerID="7754c8fe9636c45be48540d220378291dd092555078b713c2c7e41cef35263ba"
Oct 31 03:38:15.473252 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:15.473213    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"]
Oct 31 03:38:15.519827 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:15.519789    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-prtgn"]
Oct 31 03:38:16.178829 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:38:16.178789    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:38:16.607147 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:38:16.607111    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae6e719-0fac-495f-b792-aecd2e6fd7cc" path="/var/lib/kubelet/pods/8ae6e719-0fac-495f-b792-aecd2e6fd7cc/volumes"
Oct 31 03:38:46.176512 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:38:46.176471    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:39:09.553869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:09.553823    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"]
Oct 31 03:39:09.554418 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:09.554143    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerName="istio-proxy" containerID="cri-o://9c845cf0b04409c5dae2d87edf67fd050c9a9e8281194b50bca899ebc52b5567" gracePeriod=30
Oct 31 03:39:09.600768 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:09.600722    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"]
Oct 31 03:39:09.601016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:09.600983    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww" podUID="5b97d07f-dda8-485e-9377-61b8df3ffbb5" containerName="istio-proxy" containerID="cri-o://e1a113cc32101ba2d0018191c97f2ba662f0dd70f29dbbacb02c19015986ce9b" gracePeriod=30
Oct 31 03:39:12.380677 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:12.380632    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:39:12.381106 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:12.380698    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww" podUID="5b97d07f-dda8-485e-9377-61b8df3ffbb5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:39:13.066359 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:13.066317    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:39:13.066549 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:13.066391    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:39:14.727033 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.726993    5415 generic.go:356] "Generic (PLEG): container finished" podID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerID="9c845cf0b04409c5dae2d87edf67fd050c9a9e8281194b50bca899ebc52b5567" exitCode=0
Oct 31 03:39:14.727379 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.727060    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" event={"ID":"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980","Type":"ContainerDied","Data":"9c845cf0b04409c5dae2d87edf67fd050c9a9e8281194b50bca899ebc52b5567"}
Oct 31 03:39:14.728820 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.728798    5415 generic.go:356] "Generic (PLEG): container finished" podID="5b97d07f-dda8-485e-9377-61b8df3ffbb5" containerID="e1a113cc32101ba2d0018191c97f2ba662f0dd70f29dbbacb02c19015986ce9b" exitCode=0
Oct 31 03:39:14.728964 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.728888    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww" event={"ID":"5b97d07f-dda8-485e-9377-61b8df3ffbb5","Type":"ContainerDied","Data":"e1a113cc32101ba2d0018191c97f2ba662f0dd70f29dbbacb02c19015986ce9b"}
Oct 31 03:39:14.890176 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.890148    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:39:14.916202 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.916171    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:39:14.957182 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.957149    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-envoy\") pod \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") "
Oct 31 03:39:14.957398 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.957210    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-certs\") pod \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") "
Oct 31 03:39:14.957398 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.957262    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-data\") pod \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") "
Oct 31 03:39:14.957398 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.957298    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-kube-api-access-qgc54\" (UniqueName: \"kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-kube-api-access-qgc54\") pod \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") "
Oct 31 03:39:14.957398 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.957315    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-credential-socket\") pod \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") "
Oct 31 03:39:14.957398 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.957374    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-socket\") pod \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") "
Oct 31 03:39:14.957686 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.957509    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-certs" pod "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" (UID: "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:14.957686 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.957532    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-token\" (UniqueName: \"kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-token\") pod \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") "
Oct 31 03:39:14.957686 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.957566    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-data" pod "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" (UID: "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:14.957686 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.957632    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-podinfo\") pod \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") "
Oct 31 03:39:14.957686 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.957679    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istiod-ca-cert\") pod \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\" (UID: \"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980\") "
Oct 31 03:39:14.957686 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.957660    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-credential-socket" pod "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" (UID: "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:14.958010 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.957718    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-socket" pod "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" (UID: "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:14.958162 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.958139    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istiod-ca-cert" pod "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" (UID: "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:14.958233 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.958194    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:14.958233 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.958214    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:14.958233 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.958227    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:14.958398 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.958241    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:14.959631 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.959602    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-token" pod "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" (UID: "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:14.959735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.959650    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-envoy" pod "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" (UID: "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:14.959925 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.959861    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-kube-api-access-qgc54" pod "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" (UID: "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980"). InnerVolumeSpecName "kube-api-access-qgc54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:14.960005 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:14.959921    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-podinfo" pod "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" (UID: "1ba281b6-72bf-46f8-b5ed-c1c10f1bc980"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:39:15.059003 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.058971    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-socket\") pod \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") "
Oct 31 03:39:15.059203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.059020    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-certs\") pod \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") "
Oct 31 03:39:15.059203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.059104    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-data\") pod \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") "
Oct 31 03:39:15.059203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.059147    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istiod-ca-cert\") pod \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") "
Oct 31 03:39:15.059354 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.059226    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-envoy\") pod \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") "
Oct 31 03:39:15.059354 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.059265    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-credential-socket\") pod \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") "
Oct 31 03:39:15.059354 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.059322    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-podinfo\") pod \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") "
Oct 31 03:39:15.059354 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.059329    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-socket" pod "5b97d07f-dda8-485e-9377-61b8df3ffbb5" (UID: "5b97d07f-dda8-485e-9377-61b8df3ffbb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:15.059354 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.059346    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-certs" pod "5b97d07f-dda8-485e-9377-61b8df3ffbb5" (UID: "5b97d07f-dda8-485e-9377-61b8df3ffbb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:15.059575 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.059358    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-token\" (UniqueName: \"kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-token\") pod \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") "
Oct 31 03:39:15.059575 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.059522    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-credential-socket" pod "5b97d07f-dda8-485e-9377-61b8df3ffbb5" (UID: "5b97d07f-dda8-485e-9377-61b8df3ffbb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:15.059652 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.059568    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istiod-ca-cert" pod "5b97d07f-dda8-485e-9377-61b8df3ffbb5" (UID: "5b97d07f-dda8-485e-9377-61b8df3ffbb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:15.059652 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.059618    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-kube-api-access-4jbvp\" (UniqueName: \"kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-kube-api-access-4jbvp\") pod \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\" (UID: \"5b97d07f-dda8-485e-9377-61b8df3ffbb5\") "
Oct 31 03:39:15.060096 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.060073    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgc54\" (UniqueName: \"kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-kube-api-access-qgc54\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.060096 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.060093    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.060283 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.060104    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.060283 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.060113    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.060283 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.060122    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.060283 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.060133    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.060283 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.060146    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.060283 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.060160    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.060283 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.060169    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.060680 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.060485    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-data" pod "5b97d07f-dda8-485e-9377-61b8df3ffbb5" (UID: "5b97d07f-dda8-485e-9377-61b8df3ffbb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:15.061551 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.061524    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-envoy" pod "5b97d07f-dda8-485e-9377-61b8df3ffbb5" (UID: "5b97d07f-dda8-485e-9377-61b8df3ffbb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:15.061827 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.061800    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-token" pod "5b97d07f-dda8-485e-9377-61b8df3ffbb5" (UID: "5b97d07f-dda8-485e-9377-61b8df3ffbb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:15.061978 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.061909    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-kube-api-access-4jbvp" pod "5b97d07f-dda8-485e-9377-61b8df3ffbb5" (UID: "5b97d07f-dda8-485e-9377-61b8df3ffbb5"). InnerVolumeSpecName "kube-api-access-4jbvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:15.062329 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.062309    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-podinfo" pod "5b97d07f-dda8-485e-9377-61b8df3ffbb5" (UID: "5b97d07f-dda8-485e-9377-61b8df3ffbb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:39:15.165540 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.161908    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.165540 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.161944    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.165540 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.161977    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.165540 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.161994    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jbvp\" (UniqueName: \"kubernetes.io/projected/5b97d07f-dda8-485e-9377-61b8df3ffbb5-kube-api-access-4jbvp\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.165540 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.162008    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b97d07f-dda8-485e-9377-61b8df3ffbb5-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:15.736177 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.736142    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"
Oct 31 03:39:15.736177 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.736152    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj" event={"ID":"1ba281b6-72bf-46f8-b5ed-c1c10f1bc980","Type":"ContainerDied","Data":"b257e4a76da34e9a038d55c96aa6c35dd8bac53da589345ac89424e64ae56f91"}
Oct 31 03:39:15.736649 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.736224    5415 scope.go:122] "RemoveContainer" containerID="9c845cf0b04409c5dae2d87edf67fd050c9a9e8281194b50bca899ebc52b5567"
Oct 31 03:39:15.738001 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.737969    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww" event={"ID":"5b97d07f-dda8-485e-9377-61b8df3ffbb5","Type":"ContainerDied","Data":"5aba828de417b7645d1872e4e18ccfba70a8c144b20d0ebaaa8853d49d4db153"}
Oct 31 03:39:15.738108 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.737988    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"
Oct 31 03:39:15.747596 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.747575    5415 scope.go:122] "RemoveContainer" containerID="e1a113cc32101ba2d0018191c97f2ba662f0dd70f29dbbacb02c19015986ce9b"
Oct 31 03:39:15.891813 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.891777    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"]
Oct 31 03:39:15.895382 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:15.895351    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zw2ww"]
Oct 31 03:39:16.001663 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:16.001623    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"]
Oct 31 03:39:16.008319 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:16.008288    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-pl5mj"]
Oct 31 03:39:16.178496 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:39:16.178413    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:39:16.606541 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:16.606505    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" path="/var/lib/kubelet/pods/1ba281b6-72bf-46f8-b5ed-c1c10f1bc980/volumes"
Oct 31 03:39:16.607127 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:16.607071    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b97d07f-dda8-485e-9377-61b8df3ffbb5" path="/var/lib/kubelet/pods/5b97d07f-dda8-485e-9377-61b8df3ffbb5/volumes"
Oct 31 03:39:46.176891 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:39:46.176840    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:39:50.502208 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:50.502169    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"]
Oct 31 03:39:50.502673 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:50.502437    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb" podUID="8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" containerName="istio-proxy" containerID="cri-o://4f2a80a2c84d1b09e5d8c372a02ee04449c5ebb46e5c6eb781484d66affe746b" gracePeriod=30
Oct 31 03:39:52.864643 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:52.864595    5415 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:39:52.865166 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:52.864687    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb" podUID="8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:39:55.769544 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.769481    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:39:55.927405 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.927364    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-certs\") pod \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") "
Oct 31 03:39:55.927598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.927415    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-token\" (UniqueName: \"kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-token\") pod \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") "
Oct 31 03:39:55.927598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.927476    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-data\") pod \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") "
Oct 31 03:39:55.927598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.927519    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-envoy\") pod \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") "
Oct 31 03:39:55.927598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.927594    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-socket\") pod \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") "
Oct 31 03:39:55.927830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.927606    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-certs" pod "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" (UID: "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.927830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.927663    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-kube-api-access-fxj4s\" (UniqueName: \"kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-kube-api-access-fxj4s\") pod \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") "
Oct 31 03:39:55.927830 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.927757    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-podinfo\") pod \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") "
Oct 31 03:39:55.928028 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.927834    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-credential-socket\") pod \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") "
Oct 31 03:39:55.928028 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.927924    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-data" pod "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" (UID: "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.928028 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.927945    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-socket" pod "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" (UID: "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.928028 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.927930    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istiod-ca-cert\") pod \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\" (UID: \"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0\") "
Oct 31 03:39:55.928250 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.928227    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istiod-ca-cert" pod "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" (UID: "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:55.928494 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.928458    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-credential-socket" pod "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" (UID: "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.929615 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.929575    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.929615 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.929601    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.930028 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.929617    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.930028 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.929994    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.930304 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.930276    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.930573 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.930535    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-token" pod "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" (UID: "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:55.931190 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.931163    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-kube-api-access-fxj4s" pod "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" (UID: "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0"). InnerVolumeSpecName "kube-api-access-fxj4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:55.931437 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.931413    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-envoy" pod "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" (UID: "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.932227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:55.932200    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-podinfo" pod "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" (UID: "8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:39:56.031572 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.031481    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.031572 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.031519    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.031572 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.031534    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxj4s\" (UniqueName: \"kubernetes.io/projected/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-kube-api-access-fxj4s\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.031572 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.031549    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.035503 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.035476    5415 generic.go:356] "Generic (PLEG): container finished" podID="8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" containerID="4f2a80a2c84d1b09e5d8c372a02ee04449c5ebb46e5c6eb781484d66affe746b" exitCode=0
Oct 31 03:39:56.035653 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.035598    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"
Oct 31 03:39:56.035693 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.035605    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb" event={"ID":"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0","Type":"ContainerDied","Data":"4f2a80a2c84d1b09e5d8c372a02ee04449c5ebb46e5c6eb781484d66affe746b"}
Oct 31 03:39:56.035730 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.035719    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb" event={"ID":"8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0","Type":"ContainerDied","Data":"2e1d211e5c0b6dfcf6b60874a79e517f25bf05e5378deb61bf6da858904215a0"}
Oct 31 03:39:56.035766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.035741    5415 scope.go:122] "RemoveContainer" containerID="4f2a80a2c84d1b09e5d8c372a02ee04449c5ebb46e5c6eb781484d66affe746b"
Oct 31 03:39:56.047201 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.047176    5415 scope.go:122] "RemoveContainer" containerID="4f2a80a2c84d1b09e5d8c372a02ee04449c5ebb46e5c6eb781484d66affe746b"
Oct 31 03:39:56.047492 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:39:56.047470    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2a80a2c84d1b09e5d8c372a02ee04449c5ebb46e5c6eb781484d66affe746b\": container with ID starting with 4f2a80a2c84d1b09e5d8c372a02ee04449c5ebb46e5c6eb781484d66affe746b not found: ID does not exist" containerID="4f2a80a2c84d1b09e5d8c372a02ee04449c5ebb46e5c6eb781484d66affe746b"
Oct 31 03:39:56.047571 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.047502    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2a80a2c84d1b09e5d8c372a02ee04449c5ebb46e5c6eb781484d66affe746b"} err="failed to get container status \"4f2a80a2c84d1b09e5d8c372a02ee04449c5ebb46e5c6eb781484d66affe746b\": rpc error: code = NotFound desc = could not find container \"4f2a80a2c84d1b09e5d8c372a02ee04449c5ebb46e5c6eb781484d66affe746b\": container with ID starting with 4f2a80a2c84d1b09e5d8c372a02ee04449c5ebb46e5c6eb781484d66affe746b not found: ID does not exist"
Oct 31 03:39:56.191626 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.191591    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"]
Oct 31 03:39:56.197579 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.197544    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-r49pb"]
Oct 31 03:39:56.606009 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:39:56.605974    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" path="/var/lib/kubelet/pods/8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0/volumes"
Oct 31 03:40:11.152656 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.152620    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"]
Oct 31 03:40:11.153125 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.152840    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx" podUID="3da717ff-7f08-4393-849b-052cca9e3680" containerName="maas-api" containerID="cri-o://f132450985282b77e355c0d46cae2425b7aa6ab9a511c62013a00b3004cd0253" gracePeriod=30
Oct 31 03:40:11.473464 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.473428    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:40:11.517842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.517613    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-maas-api-tls\") pod \"3da717ff-7f08-4393-849b-052cca9e3680\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") "
Oct 31 03:40:11.517842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.517693    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da717ff-7f08-4393-849b-052cca9e3680-kube-api-access-vh2pz\" (UniqueName: \"kubernetes.io/projected/3da717ff-7f08-4393-849b-052cca9e3680-kube-api-access-vh2pz\") pod \"3da717ff-7f08-4393-849b-052cca9e3680\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") "
Oct 31 03:40:11.517842 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.517755    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da717ff-7f08-4393-849b-052cca9e3680-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da717ff-7f08-4393-849b-052cca9e3680-openshift-service-ca\") pod \"3da717ff-7f08-4393-849b-052cca9e3680\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") "
Oct 31 03:40:11.518171 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.517909    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-metrics-certs\") pod \"3da717ff-7f08-4393-849b-052cca9e3680\" (UID: \"3da717ff-7f08-4393-849b-052cca9e3680\") "
Oct 31 03:40:11.518592 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.518548    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da717ff-7f08-4393-849b-052cca9e3680-openshift-service-ca" pod "3da717ff-7f08-4393-849b-052cca9e3680" (UID: "3da717ff-7f08-4393-849b-052cca9e3680"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:40:11.520438 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.520252    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-maas-api-tls" pod "3da717ff-7f08-4393-849b-052cca9e3680" (UID: "3da717ff-7f08-4393-849b-052cca9e3680"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:40:11.520798 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.520774    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da717ff-7f08-4393-849b-052cca9e3680-kube-api-access-vh2pz" pod "3da717ff-7f08-4393-849b-052cca9e3680" (UID: "3da717ff-7f08-4393-849b-052cca9e3680"). InnerVolumeSpecName "kube-api-access-vh2pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:11.521353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.521314    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-metrics-certs" pod "3da717ff-7f08-4393-849b-052cca9e3680" (UID: "3da717ff-7f08-4393-849b-052cca9e3680"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:40:11.619735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.619697    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:11.619735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.619729    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh2pz\" (UniqueName: \"kubernetes.io/projected/3da717ff-7f08-4393-849b-052cca9e3680-kube-api-access-vh2pz\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:11.619735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.619742    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da717ff-7f08-4393-849b-052cca9e3680-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:11.620012 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:11.619751    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3da717ff-7f08-4393-849b-052cca9e3680-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:12.132847 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:12.132813    5415 generic.go:356] "Generic (PLEG): container finished" podID="3da717ff-7f08-4393-849b-052cca9e3680" containerID="f132450985282b77e355c0d46cae2425b7aa6ab9a511c62013a00b3004cd0253" exitCode=0
Oct 31 03:40:12.133039 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:12.132854    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"
Oct 31 03:40:12.133039 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:12.132914    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx" event={"ID":"3da717ff-7f08-4393-849b-052cca9e3680","Type":"ContainerDied","Data":"f132450985282b77e355c0d46cae2425b7aa6ab9a511c62013a00b3004cd0253"}
Oct 31 03:40:12.133039 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:12.132982    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx" event={"ID":"3da717ff-7f08-4393-849b-052cca9e3680","Type":"ContainerDied","Data":"e956abf51a35eb703038d9895dee66a432b4582d32ca8be0f1715f8ff249d729"}
Oct 31 03:40:12.133039 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:12.133012    5415 scope.go:122] "RemoveContainer" containerID="f132450985282b77e355c0d46cae2425b7aa6ab9a511c62013a00b3004cd0253"
Oct 31 03:40:12.143292 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:12.143268    5415 scope.go:122] "RemoveContainer" containerID="f132450985282b77e355c0d46cae2425b7aa6ab9a511c62013a00b3004cd0253"
Oct 31 03:40:12.143577 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:40:12.143558    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f132450985282b77e355c0d46cae2425b7aa6ab9a511c62013a00b3004cd0253\": container with ID starting with f132450985282b77e355c0d46cae2425b7aa6ab9a511c62013a00b3004cd0253 not found: ID does not exist" containerID="f132450985282b77e355c0d46cae2425b7aa6ab9a511c62013a00b3004cd0253"
Oct 31 03:40:12.143624 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:12.143590    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f132450985282b77e355c0d46cae2425b7aa6ab9a511c62013a00b3004cd0253"} err="failed to get container status \"f132450985282b77e355c0d46cae2425b7aa6ab9a511c62013a00b3004cd0253\": rpc error: code = NotFound desc = could not find container \"f132450985282b77e355c0d46cae2425b7aa6ab9a511c62013a00b3004cd0253\": container with ID starting with f132450985282b77e355c0d46cae2425b7aa6ab9a511c62013a00b3004cd0253 not found: ID does not exist"
Oct 31 03:40:12.319911 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:12.307380    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"]
Oct 31 03:40:12.380629 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:12.380560    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d3964-9875fb87d-gl5xx"]
Oct 31 03:40:12.605903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:12.605849    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da717ff-7f08-4393-849b-052cca9e3680" path="/var/lib/kubelet/pods/3da717ff-7f08-4393-849b-052cca9e3680/volumes"
Oct 31 03:40:16.175441 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:40:16.175401    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:40:43.584253 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.584208    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"]
Oct 31 03:40:43.584988 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.584967    5415 cpu_manager.go:414] "containerMap: removing container" podUID="3da717ff-7f08-4393-849b-052cca9e3680" containerName="maas-api"
Oct 31 03:40:43.584988 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.584988    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da717ff-7f08-4393-849b-052cca9e3680" containerName="maas-api"
Oct 31 03:40:43.585156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.585010    5415 cpu_manager.go:414] "containerMap: removing container" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerName="istio-proxy"
Oct 31 03:40:43.585156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.585017    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerName="istio-proxy"
Oct 31 03:40:43.585156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.585027    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8ae6e719-0fac-495f-b792-aecd2e6fd7cc" containerName="istio-proxy"
Oct 31 03:40:43.585156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.585066    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae6e719-0fac-495f-b792-aecd2e6fd7cc" containerName="istio-proxy"
Oct 31 03:40:43.585156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.585080    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" containerName="istio-proxy"
Oct 31 03:40:43.585156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.585087    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" containerName="istio-proxy"
Oct 31 03:40:43.585156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.585108    5415 cpu_manager.go:414] "containerMap: removing container" podUID="5b97d07f-dda8-485e-9377-61b8df3ffbb5" containerName="istio-proxy"
Oct 31 03:40:43.585156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.585115    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b97d07f-dda8-485e-9377-61b8df3ffbb5" containerName="istio-proxy"
Oct 31 03:40:43.585535 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.585347    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b97d07f-dda8-485e-9377-61b8df3ffbb5" containerName="istio-proxy"
Oct 31 03:40:43.585535 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.585374    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da717ff-7f08-4393-849b-052cca9e3680" containerName="maas-api"
Oct 31 03:40:43.585535 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.585386    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba281b6-72bf-46f8-b5ed-c1c10f1bc980" containerName="istio-proxy"
Oct 31 03:40:43.585535 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.585394    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae6e719-0fac-495f-b792-aecd2e6fd7cc" containerName="istio-proxy"
Oct 31 03:40:43.585535 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.585405    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fcb6dd5-4b5c-4aaf-83ce-9f4fd63ad4b0" containerName="istio-proxy"
Oct 31 03:40:43.590236 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.590213    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"
Oct 31 03:40:43.650052 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.650019    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c561e3-dockercfg-mdsj6\""
Oct 31 03:40:43.658597 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.658569    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"]
Oct 31 03:40:43.675866 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.675837    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c561e3\""
Oct 31 03:40:43.718989 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.718951    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"]
Oct 31 03:40:43.726079 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.726049    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"
Oct 31 03:40:43.751555 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.751508    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9f51f4-cb08-4e82-b8d4-00694552fd27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr\" (UID: \"0e9f51f4-cb08-4e82-b8d4-00694552fd27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"
Oct 31 03:40:43.754161 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.754132    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfdkx\" (UniqueName: \"kubernetes.io/projected/0e9f51f4-cb08-4e82-b8d4-00694552fd27-kube-api-access-nfdkx\") pod \"payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr\" (UID: \"0e9f51f4-cb08-4e82-b8d4-00694552fd27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"
Oct 31 03:40:43.781687 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.781652    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"]
Oct 31 03:40:43.857030 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.856936    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfdkx\" (UniqueName: \"kubernetes.io/projected/0e9f51f4-cb08-4e82-b8d4-00694552fd27-kube-api-access-nfdkx\") pod \"payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr\" (UID: \"0e9f51f4-cb08-4e82-b8d4-00694552fd27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"
Oct 31 03:40:43.857030 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.857009    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj\" (UID: \"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"
Oct 31 03:40:43.857257 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.857065    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h76q\" (UniqueName: \"kubernetes.io/projected/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-kube-api-access-2h76q\") pod \"payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj\" (UID: \"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"
Oct 31 03:40:43.857257 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.857192    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9f51f4-cb08-4e82-b8d4-00694552fd27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr\" (UID: \"0e9f51f4-cb08-4e82-b8d4-00694552fd27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"
Oct 31 03:40:43.857936 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.857910    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9f51f4-cb08-4e82-b8d4-00694552fd27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr\" (UID: \"0e9f51f4-cb08-4e82-b8d4-00694552fd27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"
Oct 31 03:40:43.911024 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.910992    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfdkx\" (UniqueName: \"kubernetes.io/projected/0e9f51f4-cb08-4e82-b8d4-00694552fd27-kube-api-access-nfdkx\") pod \"payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr\" (UID: \"0e9f51f4-cb08-4e82-b8d4-00694552fd27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"
Oct 31 03:40:43.958030 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.957991    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj\" (UID: \"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"
Oct 31 03:40:43.958229 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.958064    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h76q\" (UniqueName: \"kubernetes.io/projected/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-kube-api-access-2h76q\") pod \"payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj\" (UID: \"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"
Oct 31 03:40:43.958720 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:43.958699    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj\" (UID: \"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"
Oct 31 03:40:44.012032 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:44.011995    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h76q\" (UniqueName: \"kubernetes.io/projected/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-kube-api-access-2h76q\") pod \"payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj\" (UID: \"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"
Oct 31 03:40:44.042822 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:44.042787    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"
Oct 31 03:40:44.203390 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:44.203301    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"
Oct 31 03:40:44.411902 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:44.411861    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"]
Oct 31 03:40:44.414312 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:40:44.414279    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6d4ed8_fc33_407d_82ee_03cd5f4cb22a.slice/crio-a3daaf708802585419f56bde932e5a7b6e119fbc282534b8c15419b9d00c7f5a WatchSource:0}: Error finding container a3daaf708802585419f56bde932e5a7b6e119fbc282534b8c15419b9d00c7f5a: Status 404 returned error can't find the container with id a3daaf708802585419f56bde932e5a7b6e119fbc282534b8c15419b9d00c7f5a
Oct 31 03:40:44.580716 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:44.580690    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"]
Oct 31 03:40:44.582374 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:40:44.582345    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9f51f4_cb08_4e82_b8d4_00694552fd27.slice/crio-28d6c5c6c1c9f3e0bbcfde401bf6a5c5d92773ca649fb2a49ea60f9fe63b7d6d WatchSource:0}: Error finding container 28d6c5c6c1c9f3e0bbcfde401bf6a5c5d92773ca649fb2a49ea60f9fe63b7d6d: Status 404 returned error can't find the container with id 28d6c5c6c1c9f3e0bbcfde401bf6a5c5d92773ca649fb2a49ea60f9fe63b7d6d
Oct 31 03:40:45.340447 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:45.340365    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr" event={"ID":"0e9f51f4-cb08-4e82-b8d4-00694552fd27","Type":"ContainerStarted","Data":"d4f85c053ec7f599f52a09a525b63d0037d3752cf31b586f5173b6c7530c011b"}
Oct 31 03:40:45.340447 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:45.340417    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr" event={"ID":"0e9f51f4-cb08-4e82-b8d4-00694552fd27","Type":"ContainerStarted","Data":"28d6c5c6c1c9f3e0bbcfde401bf6a5c5d92773ca649fb2a49ea60f9fe63b7d6d"}
Oct 31 03:40:45.340955 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:45.340543    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"
Oct 31 03:40:45.342202 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:45.342173    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj" event={"ID":"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a","Type":"ContainerStarted","Data":"5ac545eb5eeed9d76504067f4a2a68a3c47d534c7ff0964d2f082544d03614d6"}
Oct 31 03:40:45.342323 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:45.342225    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj" event={"ID":"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a","Type":"ContainerStarted","Data":"a3daaf708802585419f56bde932e5a7b6e119fbc282534b8c15419b9d00c7f5a"}
Oct 31 03:40:45.342323 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:45.342257    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"
Oct 31 03:40:45.449963 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:45.449907    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr" podStartSLOduration=2.44989192 podStartE2EDuration="2.44989192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:45.448055626 +0000 UTC m=+1987.008535361" watchObservedRunningTime="2026-10-31 03:40:45.44989192 +0000 UTC m=+1987.010371693"
Oct 31 03:40:45.557164 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:45.557105    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj" podStartSLOduration=2.557090045 podStartE2EDuration="2.557090045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:45.554427055 +0000 UTC m=+1987.114906808" watchObservedRunningTime="2026-10-31 03:40:45.557090045 +0000 UTC m=+1987.117569793"
Oct 31 03:40:46.175449 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:40:46.175409    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:40:56.351409 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:56.351379    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"
Oct 31 03:40:56.351774 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:40:56.351450    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"
Oct 31 03:41:16.197922 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:41:16.197245    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:41:46.173823 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:41:46.173784    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:42:01.793067 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:01.793026    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"]
Oct 31 03:42:01.793588 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:01.793331    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" podUID="057c4b5b-d48f-413f-9a9c-43db3e06f380" containerName="istio-proxy" containerID="cri-o://270f6af21b472f620d09a785077c5df3ae4e4dcf1dcc6335dbcd3b80dad15637" gracePeriod=30
Oct 31 03:42:01.839686 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:01.839644    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"]
Oct 31 03:42:01.839870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:01.839855    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" podUID="a2161a56-053a-4467-8a42-608e85bcedb9" containerName="istio-proxy" containerID="cri-o://91595f7a2f621bd9533a6bf34679350961ed43bb20bc661b55af6a1876367703" gracePeriod=30
Oct 31 03:42:02.620353 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:02.620308    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-76m8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:42:02.620548 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:02.620369    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" podUID="057c4b5b-d48f-413f-9a9c-43db3e06f380" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:42:03.282258 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:03.282216    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-flbbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:42:03.282656 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:03.282273    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" podUID="a2161a56-053a-4467-8a42-608e85bcedb9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:42:07.136638 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.136610    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:42:07.179826 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.179788    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:42:07.218501 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218465    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-data\") pod \"057c4b5b-d48f-413f-9a9c-43db3e06f380\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") "
Oct 31 03:42:07.218501 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218509    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2161a56-053a-4467-8a42-608e85bcedb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2161a56-053a-4467-8a42-608e85bcedb9-istiod-ca-cert\") pod \"a2161a56-053a-4467-8a42-608e85bcedb9\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") "
Oct 31 03:42:07.218776 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218534    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-kube-api-access-9t2r6\" (UniqueName: \"kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-kube-api-access-9t2r6\") pod \"a2161a56-053a-4467-8a42-608e85bcedb9\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") "
Oct 31 03:42:07.218776 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218555    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-credential-socket\") pod \"a2161a56-053a-4467-8a42-608e85bcedb9\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") "
Oct 31 03:42:07.218776 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218635    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-certs\") pod \"057c4b5b-d48f-413f-9a9c-43db3e06f380\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") "
Oct 31 03:42:07.218776 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218669    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-data\") pod \"a2161a56-053a-4467-8a42-608e85bcedb9\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") "
Oct 31 03:42:07.218776 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218723    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-kube-api-access-ncl7d\" (UniqueName: \"kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-kube-api-access-ncl7d\") pod \"057c4b5b-d48f-413f-9a9c-43db3e06f380\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") "
Oct 31 03:42:07.218776 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218767    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-socket\") pod \"a2161a56-053a-4467-8a42-608e85bcedb9\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") "
Oct 31 03:42:07.219118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218796    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-istio-token\" (UniqueName: \"kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-istio-token\") pod \"a2161a56-053a-4467-8a42-608e85bcedb9\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") "
Oct 31 03:42:07.219118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218835    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-credential-socket\") pod \"057c4b5b-d48f-413f-9a9c-43db3e06f380\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") "
Oct 31 03:42:07.219118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218838    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-data" pod "057c4b5b-d48f-413f-9a9c-43db3e06f380" (UID: "057c4b5b-d48f-413f-9a9c-43db3e06f380"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:07.219118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218939    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-credential-socket" pod "a2161a56-053a-4467-8a42-608e85bcedb9" (UID: "a2161a56-053a-4467-8a42-608e85bcedb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:07.219118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218945    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-token\" (UniqueName: \"kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-token\") pod \"057c4b5b-d48f-413f-9a9c-43db3e06f380\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") "
Oct 31 03:42:07.219118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.218967    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2161a56-053a-4467-8a42-608e85bcedb9-istiod-ca-cert" pod "a2161a56-053a-4467-8a42-608e85bcedb9" (UID: "a2161a56-053a-4467-8a42-608e85bcedb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:42:07.219118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.219011    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2161a56-053a-4467-8a42-608e85bcedb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2161a56-053a-4467-8a42-608e85bcedb9-istio-podinfo\") pod \"a2161a56-053a-4467-8a42-608e85bcedb9\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") "
Oct 31 03:42:07.219118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.219044    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-podinfo\") pod \"057c4b5b-d48f-413f-9a9c-43db3e06f380\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") "
Oct 31 03:42:07.219118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.219109    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-socket\") pod \"057c4b5b-d48f-413f-9a9c-43db3e06f380\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") "
Oct 31 03:42:07.219591 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.219179    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057c4b5b-d48f-413f-9a9c-43db3e06f380-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/057c4b5b-d48f-413f-9a9c-43db3e06f380-istiod-ca-cert\") pod \"057c4b5b-d48f-413f-9a9c-43db3e06f380\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") "
Oct 31 03:42:07.219591 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.219220    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-envoy\") pod \"a2161a56-053a-4467-8a42-608e85bcedb9\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") "
Oct 31 03:42:07.219591 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.219261    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-data" pod "a2161a56-053a-4467-8a42-608e85bcedb9" (UID: "a2161a56-053a-4467-8a42-608e85bcedb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:07.219591 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.219260    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-certs" pod "057c4b5b-d48f-413f-9a9c-43db3e06f380" (UID: "057c4b5b-d48f-413f-9a9c-43db3e06f380"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:07.219591 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.219306    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-certs\") pod \"a2161a56-053a-4467-8a42-608e85bcedb9\" (UID: \"a2161a56-053a-4467-8a42-608e85bcedb9\") "
Oct 31 03:42:07.219591 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.219353    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-envoy\") pod \"057c4b5b-d48f-413f-9a9c-43db3e06f380\" (UID: \"057c4b5b-d48f-413f-9a9c-43db3e06f380\") "
Oct 31 03:42:07.219591 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.219550    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-socket" pod "a2161a56-053a-4467-8a42-608e85bcedb9" (UID: "a2161a56-053a-4467-8a42-608e85bcedb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:07.221435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.220130    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.221435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.220165    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.221435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.220182    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2161a56-053a-4467-8a42-608e85bcedb9-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.221435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.220198    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.221435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.220213    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.221435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.220227    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.221435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.220520    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-credential-socket" pod "057c4b5b-d48f-413f-9a9c-43db3e06f380" (UID: "057c4b5b-d48f-413f-9a9c-43db3e06f380"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:07.221435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.220808    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057c4b5b-d48f-413f-9a9c-43db3e06f380-istiod-ca-cert" pod "057c4b5b-d48f-413f-9a9c-43db3e06f380" (UID: "057c4b5b-d48f-413f-9a9c-43db3e06f380"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:42:07.221435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.221143    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-socket" pod "057c4b5b-d48f-413f-9a9c-43db3e06f380" (UID: "057c4b5b-d48f-413f-9a9c-43db3e06f380"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:07.222121 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.221513    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-kube-api-access-9t2r6" pod "a2161a56-053a-4467-8a42-608e85bcedb9" (UID: "a2161a56-053a-4467-8a42-608e85bcedb9"). InnerVolumeSpecName "kube-api-access-9t2r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:07.222121 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.221553    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-token" pod "057c4b5b-d48f-413f-9a9c-43db3e06f380" (UID: "057c4b5b-d48f-413f-9a9c-43db3e06f380"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:07.222121 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.221615    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-istio-token" pod "a2161a56-053a-4467-8a42-608e85bcedb9" (UID: "a2161a56-053a-4467-8a42-608e85bcedb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:07.222121 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.221713    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2161a56-053a-4467-8a42-608e85bcedb9-istio-podinfo" pod "a2161a56-053a-4467-8a42-608e85bcedb9" (UID: "a2161a56-053a-4467-8a42-608e85bcedb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:42:07.222121 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.221786    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-certs" pod "a2161a56-053a-4467-8a42-608e85bcedb9" (UID: "a2161a56-053a-4467-8a42-608e85bcedb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:07.222121 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.221850    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-envoy" pod "057c4b5b-d48f-413f-9a9c-43db3e06f380" (UID: "057c4b5b-d48f-413f-9a9c-43db3e06f380"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:07.222121 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.222121    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-kube-api-access-ncl7d" pod "057c4b5b-d48f-413f-9a9c-43db3e06f380" (UID: "057c4b5b-d48f-413f-9a9c-43db3e06f380"). InnerVolumeSpecName "kube-api-access-ncl7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:07.222511 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.222151    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-podinfo" pod "057c4b5b-d48f-413f-9a9c-43db3e06f380" (UID: "057c4b5b-d48f-413f-9a9c-43db3e06f380"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:42:07.223274 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.223255    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-envoy" pod "a2161a56-053a-4467-8a42-608e85bcedb9" (UID: "a2161a56-053a-4467-8a42-608e85bcedb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:07.322110 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.322065    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncl7d\" (UniqueName: \"kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-kube-api-access-ncl7d\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.322110 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.322113    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.322364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.322129    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.322364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.322145    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.322364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.322159    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2161a56-053a-4467-8a42-608e85bcedb9-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.322364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.322174    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.322364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.322187    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.322364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.322213    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/057c4b5b-d48f-413f-9a9c-43db3e06f380-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.322364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.322227    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.322364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.322240    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2161a56-053a-4467-8a42-608e85bcedb9-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.322364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.322254    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/057c4b5b-d48f-413f-9a9c-43db3e06f380-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.322364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.322270    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t2r6\" (UniqueName: \"kubernetes.io/projected/a2161a56-053a-4467-8a42-608e85bcedb9-kube-api-access-9t2r6\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:07.550579 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.550488    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"]
Oct 31 03:42:07.550852 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.550711    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" podUID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" containerName="istio-proxy" containerID="cri-o://6d89376a0bad26cb07c7eb8aed302741ffb3247cb35060e9795f37d5dc0bc5ab" gracePeriod=30
Oct 31 03:42:07.861475 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.861432    5415 generic.go:356] "Generic (PLEG): container finished" podID="a2161a56-053a-4467-8a42-608e85bcedb9" containerID="91595f7a2f621bd9533a6bf34679350961ed43bb20bc661b55af6a1876367703" exitCode=0
Oct 31 03:42:07.861685 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.861489    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"
Oct 31 03:42:07.861685 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.861516    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" event={"ID":"a2161a56-053a-4467-8a42-608e85bcedb9","Type":"ContainerDied","Data":"91595f7a2f621bd9533a6bf34679350961ed43bb20bc661b55af6a1876367703"}
Oct 31 03:42:07.861685 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.861586    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt" event={"ID":"a2161a56-053a-4467-8a42-608e85bcedb9","Type":"ContainerDied","Data":"586718237c1932a652f118555b8aa0af2ef2d9dc3754a88e8542b69381076068"}
Oct 31 03:42:07.861685 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.861616    5415 scope.go:122] "RemoveContainer" containerID="91595f7a2f621bd9533a6bf34679350961ed43bb20bc661b55af6a1876367703"
Oct 31 03:42:07.863432 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.863114    5415 generic.go:356] "Generic (PLEG): container finished" podID="057c4b5b-d48f-413f-9a9c-43db3e06f380" containerID="270f6af21b472f620d09a785077c5df3ae4e4dcf1dcc6335dbcd3b80dad15637" exitCode=0
Oct 31 03:42:07.863432 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.863176    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"
Oct 31 03:42:07.863432 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.863175    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" event={"ID":"057c4b5b-d48f-413f-9a9c-43db3e06f380","Type":"ContainerDied","Data":"270f6af21b472f620d09a785077c5df3ae4e4dcf1dcc6335dbcd3b80dad15637"}
Oct 31 03:42:07.863432 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.863281    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v" event={"ID":"057c4b5b-d48f-413f-9a9c-43db3e06f380","Type":"ContainerDied","Data":"821cbbd9299603b2dee74f59bb7a23960e1f2e4b6002b57e77d03cd7756a90b8"}
Oct 31 03:42:07.874489 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.874363    5415 scope.go:122] "RemoveContainer" containerID="91595f7a2f621bd9533a6bf34679350961ed43bb20bc661b55af6a1876367703"
Oct 31 03:42:07.874776 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:42:07.874755    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91595f7a2f621bd9533a6bf34679350961ed43bb20bc661b55af6a1876367703\": container with ID starting with 91595f7a2f621bd9533a6bf34679350961ed43bb20bc661b55af6a1876367703 not found: ID does not exist" containerID="91595f7a2f621bd9533a6bf34679350961ed43bb20bc661b55af6a1876367703"
Oct 31 03:42:07.874867 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.874786    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91595f7a2f621bd9533a6bf34679350961ed43bb20bc661b55af6a1876367703"} err="failed to get container status \"91595f7a2f621bd9533a6bf34679350961ed43bb20bc661b55af6a1876367703\": rpc error: code = NotFound desc = could not find container \"91595f7a2f621bd9533a6bf34679350961ed43bb20bc661b55af6a1876367703\": container with ID starting with 91595f7a2f621bd9533a6bf34679350961ed43bb20bc661b55af6a1876367703 not found: ID does not exist"
Oct 31 03:42:07.874867 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.874807    5415 scope.go:122] "RemoveContainer" containerID="270f6af21b472f620d09a785077c5df3ae4e4dcf1dcc6335dbcd3b80dad15637"
Oct 31 03:42:07.886568 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.886546    5415 scope.go:122] "RemoveContainer" containerID="270f6af21b472f620d09a785077c5df3ae4e4dcf1dcc6335dbcd3b80dad15637"
Oct 31 03:42:07.887222 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:42:07.887194    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270f6af21b472f620d09a785077c5df3ae4e4dcf1dcc6335dbcd3b80dad15637\": container with ID starting with 270f6af21b472f620d09a785077c5df3ae4e4dcf1dcc6335dbcd3b80dad15637 not found: ID does not exist" containerID="270f6af21b472f620d09a785077c5df3ae4e4dcf1dcc6335dbcd3b80dad15637"
Oct 31 03:42:07.887363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:07.887231    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270f6af21b472f620d09a785077c5df3ae4e4dcf1dcc6335dbcd3b80dad15637"} err="failed to get container status \"270f6af21b472f620d09a785077c5df3ae4e4dcf1dcc6335dbcd3b80dad15637\": rpc error: code = NotFound desc = could not find container \"270f6af21b472f620d09a785077c5df3ae4e4dcf1dcc6335dbcd3b80dad15637\": container with ID starting with 270f6af21b472f620d09a785077c5df3ae4e4dcf1dcc6335dbcd3b80dad15637 not found: ID does not exist"
Oct 31 03:42:08.018680 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:08.018642    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"]
Oct 31 03:42:08.065763 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:08.065725    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-flbbt"]
Oct 31 03:42:08.126365 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:08.126331    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"]
Oct 31 03:42:08.131717 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:08.131689    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-76m8v"]
Oct 31 03:42:08.355173 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:08.355135    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:42:08.355563 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:08.355207    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" podUID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:42:08.606255 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:08.606223    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057c4b5b-d48f-413f-9a9c-43db3e06f380" path="/var/lib/kubelet/pods/057c4b5b-d48f-413f-9a9c-43db3e06f380/volumes"
Oct 31 03:42:08.606747 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:08.606732    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2161a56-053a-4467-8a42-608e85bcedb9" path="/var/lib/kubelet/pods/a2161a56-053a-4467-8a42-608e85bcedb9/volumes"
Oct 31 03:42:09.833741 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.833697    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"]
Oct 31 03:42:09.834280 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.834265    5415 cpu_manager.go:414] "containerMap: removing container" podUID="a2161a56-053a-4467-8a42-608e85bcedb9" containerName="istio-proxy"
Oct 31 03:42:09.834331 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.834281    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2161a56-053a-4467-8a42-608e85bcedb9" containerName="istio-proxy"
Oct 31 03:42:09.834331 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.834291    5415 cpu_manager.go:414] "containerMap: removing container" podUID="057c4b5b-d48f-413f-9a9c-43db3e06f380" containerName="istio-proxy"
Oct 31 03:42:09.834331 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.834297    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057c4b5b-d48f-413f-9a9c-43db3e06f380" containerName="istio-proxy"
Oct 31 03:42:09.834422 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.834378    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2161a56-053a-4467-8a42-608e85bcedb9" containerName="istio-proxy"
Oct 31 03:42:09.834422 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.834387    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="057c4b5b-d48f-413f-9a9c-43db3e06f380" containerName="istio-proxy"
Oct 31 03:42:09.839712 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.839685    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:09.939203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.939170    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-6f9fd2-serving-cert\""
Oct 31 03:42:09.941728 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.941698    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"]
Oct 31 03:42:09.953501 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.953472    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-metrics-certs\") pod \"maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:09.953640 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.953520    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:09.953640 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.953618    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:09.953722 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:09.953679    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnb7w\" (UniqueName: \"kubernetes.io/projected/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-kube-api-access-mnb7w\") pod \"maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:10.055308 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:10.055272    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-metrics-certs\") pod \"maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:10.055488 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:10.055332    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:10.055488 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:10.055416    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:10.055665 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:42:10.055641    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-serving-cert: secret "maas-api-e2e-praxis-a-6f9fd2-serving-cert" not found
Oct 31 03:42:10.055766 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:42:10.055740    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-maas-api-tls podName:156d2f57-24b3-43a4-9a06-3bfa3a7f4df6 nodeName:}" failed. No retries permitted until 2026-10-31 03:42:10.555715811 +0000 UTC m=+2072.116195537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-maas-api-tls") pod "maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh" (UID: "156d2f57-24b3-43a4-9a06-3bfa3a7f4df6") : secret "maas-api-e2e-praxis-a-6f9fd2-serving-cert" not found
Oct 31 03:42:10.055766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:10.055646    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnb7w\" (UniqueName: \"kubernetes.io/projected/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-kube-api-access-mnb7w\") pod \"maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:10.056095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:10.056071    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:10.057794 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:10.057775    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-metrics-certs\") pod \"maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:10.113042 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:10.112969    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnb7w\" (UniqueName: \"kubernetes.io/projected/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-kube-api-access-mnb7w\") pod \"maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:10.562049 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:10.561953    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:10.564519 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:10.564494    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:10.753172 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:10.753137    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:11.123458 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:11.123433    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"]
Oct 31 03:42:11.126215 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:42:11.126184    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156d2f57_24b3_43a4_9a06_3bfa3a7f4df6.slice/crio-07a1ee2a497d744795e4f4bf04f70f96d297c865bf8be8dab5f1abb09f79a809 WatchSource:0}: Error finding container 07a1ee2a497d744795e4f4bf04f70f96d297c865bf8be8dab5f1abb09f79a809: Status 404 returned error can't find the container with id 07a1ee2a497d744795e4f4bf04f70f96d297c865bf8be8dab5f1abb09f79a809
Oct 31 03:42:11.893242 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:11.893198    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh" event={"ID":"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6","Type":"ContainerStarted","Data":"07a1ee2a497d744795e4f4bf04f70f96d297c865bf8be8dab5f1abb09f79a809"}
Oct 31 03:42:12.828055 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.827985    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:42:12.893630 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.893593    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-data\") pod \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") "
Oct 31 03:42:12.893832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.893637    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-podinfo\") pod \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") "
Oct 31 03:42:12.893832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.893693    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istiod-ca-cert\") pod \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") "
Oct 31 03:42:12.893832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.893761    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-credential-socket\") pod \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") "
Oct 31 03:42:12.893832 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.893784    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-token\" (UniqueName: \"kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-token\") pod \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") "
Oct 31 03:42:12.894079 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.893843    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-envoy\") pod \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") "
Oct 31 03:42:12.894079 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.893928    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-data" pod "51ec02cc-eaaa-4913-8904-6a97d51a0d46" (UID: "51ec02cc-eaaa-4913-8904-6a97d51a0d46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:12.894079 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.893981    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-certs\") pod \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") "
Oct 31 03:42:12.894079 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.894012    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-kube-api-access-d9647\" (UniqueName: \"kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-kube-api-access-d9647\") pod \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") "
Oct 31 03:42:12.894079 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.894064    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-socket\") pod \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\" (UID: \"51ec02cc-eaaa-4913-8904-6a97d51a0d46\") "
Oct 31 03:42:12.894336 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.894286    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-certs" pod "51ec02cc-eaaa-4913-8904-6a97d51a0d46" (UID: "51ec02cc-eaaa-4913-8904-6a97d51a0d46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:12.894336 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.894320    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istiod-ca-cert" pod "51ec02cc-eaaa-4913-8904-6a97d51a0d46" (UID: "51ec02cc-eaaa-4913-8904-6a97d51a0d46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:42:12.894449 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.894353    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-credential-socket" pod "51ec02cc-eaaa-4913-8904-6a97d51a0d46" (UID: "51ec02cc-eaaa-4913-8904-6a97d51a0d46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:12.894616 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.894596    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-socket" pod "51ec02cc-eaaa-4913-8904-6a97d51a0d46" (UID: "51ec02cc-eaaa-4913-8904-6a97d51a0d46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:12.894937 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.894919    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:12.894937 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.894936    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:12.895096 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.894946    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:12.895096 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.894959    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:12.895096 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.894973    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:12.896377 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.896334    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-envoy" pod "51ec02cc-eaaa-4913-8904-6a97d51a0d46" (UID: "51ec02cc-eaaa-4913-8904-6a97d51a0d46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:12.896464 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.896418    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-podinfo" pod "51ec02cc-eaaa-4913-8904-6a97d51a0d46" (UID: "51ec02cc-eaaa-4913-8904-6a97d51a0d46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:42:12.896464 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.896452    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-token" pod "51ec02cc-eaaa-4913-8904-6a97d51a0d46" (UID: "51ec02cc-eaaa-4913-8904-6a97d51a0d46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:12.896726 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.896702    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-kube-api-access-d9647" pod "51ec02cc-eaaa-4913-8904-6a97d51a0d46" (UID: "51ec02cc-eaaa-4913-8904-6a97d51a0d46"). InnerVolumeSpecName "kube-api-access-d9647". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:12.902007 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.901975    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh" event={"ID":"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6","Type":"ContainerStarted","Data":"d216e5e84c754fe2e60fc696888636f578a0881b31fe42e1dbfe6d95e73f430b"}
Oct 31 03:42:12.902157 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.902043    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:12.903382 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.903358    5415 generic.go:356] "Generic (PLEG): container finished" podID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" containerID="6d89376a0bad26cb07c7eb8aed302741ffb3247cb35060e9795f37d5dc0bc5ab" exitCode=0
Oct 31 03:42:12.903503 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.903405    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"
Oct 31 03:42:12.903503 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.903437    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" event={"ID":"51ec02cc-eaaa-4913-8904-6a97d51a0d46","Type":"ContainerDied","Data":"6d89376a0bad26cb07c7eb8aed302741ffb3247cb35060e9795f37d5dc0bc5ab"}
Oct 31 03:42:12.903503 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.903484    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm" event={"ID":"51ec02cc-eaaa-4913-8904-6a97d51a0d46","Type":"ContainerDied","Data":"51ebd421c132c923065a422ed903d9aa3685838a886313dc64785d07fd417c1d"}
Oct 31 03:42:12.903654 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.903511    5415 scope.go:122] "RemoveContainer" containerID="6d89376a0bad26cb07c7eb8aed302741ffb3247cb35060e9795f37d5dc0bc5ab"
Oct 31 03:42:12.923477 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.923455    5415 scope.go:122] "RemoveContainer" containerID="6d89376a0bad26cb07c7eb8aed302741ffb3247cb35060e9795f37d5dc0bc5ab"
Oct 31 03:42:12.923827 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:42:12.923797    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d89376a0bad26cb07c7eb8aed302741ffb3247cb35060e9795f37d5dc0bc5ab\": container with ID starting with 6d89376a0bad26cb07c7eb8aed302741ffb3247cb35060e9795f37d5dc0bc5ab not found: ID does not exist" containerID="6d89376a0bad26cb07c7eb8aed302741ffb3247cb35060e9795f37d5dc0bc5ab"
Oct 31 03:42:12.923903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.923839    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d89376a0bad26cb07c7eb8aed302741ffb3247cb35060e9795f37d5dc0bc5ab"} err="failed to get container status \"6d89376a0bad26cb07c7eb8aed302741ffb3247cb35060e9795f37d5dc0bc5ab\": rpc error: code = NotFound desc = could not find container \"6d89376a0bad26cb07c7eb8aed302741ffb3247cb35060e9795f37d5dc0bc5ab\": container with ID starting with 6d89376a0bad26cb07c7eb8aed302741ffb3247cb35060e9795f37d5dc0bc5ab not found: ID does not exist"
Oct 31 03:42:12.996919 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.995849    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:12.996919 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.995920    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:12.996919 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.995937    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9647\" (UniqueName: \"kubernetes.io/projected/51ec02cc-eaaa-4913-8904-6a97d51a0d46-kube-api-access-d9647\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:12.996919 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:12.995953    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ec02cc-eaaa-4913-8904-6a97d51a0d46-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:13.095009 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:13.094958    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh" podStartSLOduration=3.135660851 podStartE2EDuration="4.094944376s" totalImagesPullingTime="959.283525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:13.094168741 +0000 UTC m=+2074.654648530" watchObservedRunningTime="2026-10-31 03:42:13.094944376 +0000 UTC m=+2074.655424145"
Oct 31 03:42:13.202153 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:13.202117    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"]
Oct 31 03:42:13.206951 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:13.206866    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xffcm"]
Oct 31 03:42:14.606331 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:14.606285    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" path="/var/lib/kubelet/pods/51ec02cc-eaaa-4913-8904-6a97d51a0d46/volumes"
Oct 31 03:42:16.176425 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:42:16.176345    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:42:18.917230 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:18.917202    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:42:39.059263 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:39.059226    5415 scope.go:122] "RemoveContainer" containerID="ac8b86d5e687e91ac74eae65935238798d615b3cc4d78cdda64e5cd61e8074e0"
Oct 31 03:42:44.183861 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:44.183827    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"]
Oct 31 03:42:45.750951 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:45.750899    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"]
Oct 31 03:42:45.751491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:45.751125    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" podUID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" containerName="istio-proxy" containerID="cri-o://86cb37c0c6ca1ebb779f1c7d144e5531f613bea5d85f874ab89d48289413182f" gracePeriod=30
Oct 31 03:42:45.852106 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:45.852068    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"]
Oct 31 03:42:45.852440 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:45.852390    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg" podUID="017e3f11-748c-46c0-b54d-33112be20591" containerName="istio-proxy" containerID="cri-o://d182ca4ffd5f6689fd7eab051ff09863ca36bfa395816af241599dee3970a5d9" gracePeriod=30
Oct 31 03:42:46.176192 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:42:46.176153    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:42:47.088944 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:47.088901    5415 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:42:47.089348 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:47.088965    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" podUID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:42:48.936099 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:48.936054    5415 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:42:48.936522 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:48.936137    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg" podUID="017e3f11-748c-46c0-b54d-33112be20591" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:42:49.184896 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:49.184824    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w" podUID="1b7cef69-f9b2-407d-a283-4f292a19b91b" containerName="payload-pre-processing" containerID="cri-o://bc0569332ef44d88b31010ffc639ec19d87a1a3d8a6a873c826e764561ddd360" gracePeriod=25
Oct 31 03:42:49.444801 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:49.444727    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"
Oct 31 03:42:49.518062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:49.518019    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b7cef69-f9b2-407d-a283-4f292a19b91b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7cef69-f9b2-407d-a283-4f292a19b91b-plugins-config-volume\") pod \"1b7cef69-f9b2-407d-a283-4f292a19b91b\" (UID: \"1b7cef69-f9b2-407d-a283-4f292a19b91b\") "
Oct 31 03:42:49.518272 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:49.518248    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7cef69-f9b2-407d-a283-4f292a19b91b-kube-api-access-zfhcc\" (UniqueName: \"kubernetes.io/projected/1b7cef69-f9b2-407d-a283-4f292a19b91b-kube-api-access-zfhcc\") pod \"1b7cef69-f9b2-407d-a283-4f292a19b91b\" (UID: \"1b7cef69-f9b2-407d-a283-4f292a19b91b\") "
Oct 31 03:42:49.518471 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:49.518446    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7cef69-f9b2-407d-a283-4f292a19b91b-plugins-config-volume" pod "1b7cef69-f9b2-407d-a283-4f292a19b91b" (UID: "1b7cef69-f9b2-407d-a283-4f292a19b91b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:42:49.518690 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:49.518671    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7cef69-f9b2-407d-a283-4f292a19b91b-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:49.520375 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:49.520349    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7cef69-f9b2-407d-a283-4f292a19b91b-kube-api-access-zfhcc" pod "1b7cef69-f9b2-407d-a283-4f292a19b91b" (UID: "1b7cef69-f9b2-407d-a283-4f292a19b91b"). InnerVolumeSpecName "kube-api-access-zfhcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:49.619462 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:49.619417    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfhcc\" (UniqueName: \"kubernetes.io/projected/1b7cef69-f9b2-407d-a283-4f292a19b91b-kube-api-access-zfhcc\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:50.122552 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:50.122516    5415 generic.go:356] "Generic (PLEG): container finished" podID="1b7cef69-f9b2-407d-a283-4f292a19b91b" containerID="bc0569332ef44d88b31010ffc639ec19d87a1a3d8a6a873c826e764561ddd360" exitCode=0
Oct 31 03:42:50.123014 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:50.122564    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"
Oct 31 03:42:50.123014 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:50.122593    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w" event={"ID":"1b7cef69-f9b2-407d-a283-4f292a19b91b","Type":"ContainerDied","Data":"bc0569332ef44d88b31010ffc639ec19d87a1a3d8a6a873c826e764561ddd360"}
Oct 31 03:42:50.123014 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:50.122648    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w" event={"ID":"1b7cef69-f9b2-407d-a283-4f292a19b91b","Type":"ContainerDied","Data":"837e7a65bed46d90b631386ac665168efff0a3ec42825fba8060bca4cc026246"}
Oct 31 03:42:50.123014 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:50.122670    5415 scope.go:122] "RemoveContainer" containerID="bc0569332ef44d88b31010ffc639ec19d87a1a3d8a6a873c826e764561ddd360"
Oct 31 03:42:50.133094 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:50.133071    5415 scope.go:122] "RemoveContainer" containerID="bc0569332ef44d88b31010ffc639ec19d87a1a3d8a6a873c826e764561ddd360"
Oct 31 03:42:50.133374 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:42:50.133352    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0569332ef44d88b31010ffc639ec19d87a1a3d8a6a873c826e764561ddd360\": container with ID starting with bc0569332ef44d88b31010ffc639ec19d87a1a3d8a6a873c826e764561ddd360 not found: ID does not exist" containerID="bc0569332ef44d88b31010ffc639ec19d87a1a3d8a6a873c826e764561ddd360"
Oct 31 03:42:50.133454 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:50.133387    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0569332ef44d88b31010ffc639ec19d87a1a3d8a6a873c826e764561ddd360"} err="failed to get container status \"bc0569332ef44d88b31010ffc639ec19d87a1a3d8a6a873c826e764561ddd360\": rpc error: code = NotFound desc = could not find container \"bc0569332ef44d88b31010ffc639ec19d87a1a3d8a6a873c826e764561ddd360\": container with ID starting with bc0569332ef44d88b31010ffc639ec19d87a1a3d8a6a873c826e764561ddd360 not found: ID does not exist"
Oct 31 03:42:50.277924 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:50.277892    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"]
Oct 31 03:42:50.324893 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:50.324841    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-329d79-948cb787f-kdr5w"]
Oct 31 03:42:50.605835 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:50.605801    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7cef69-f9b2-407d-a283-4f292a19b91b" path="/var/lib/kubelet/pods/1b7cef69-f9b2-407d-a283-4f292a19b91b/volumes"
Oct 31 03:42:51.015260 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.015233    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:42:51.132060 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.132026    5415 generic.go:356] "Generic (PLEG): container finished" podID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" containerID="86cb37c0c6ca1ebb779f1c7d144e5531f613bea5d85f874ab89d48289413182f" exitCode=0
Oct 31 03:42:51.132471 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.132096    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"
Oct 31 03:42:51.132471 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.132109    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" event={"ID":"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7","Type":"ContainerDied","Data":"86cb37c0c6ca1ebb779f1c7d144e5531f613bea5d85f874ab89d48289413182f"}
Oct 31 03:42:51.132471 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.132169    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj" event={"ID":"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7","Type":"ContainerDied","Data":"eacc370b890b6e8bb09d867b2f3a3137b57724c6e4e1324671f4ab438620f560"}
Oct 31 03:42:51.132471 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.132192    5415 scope.go:122] "RemoveContainer" containerID="86cb37c0c6ca1ebb779f1c7d144e5531f613bea5d85f874ab89d48289413182f"
Oct 31 03:42:51.133716 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.133695    5415 generic.go:356] "Generic (PLEG): container finished" podID="017e3f11-748c-46c0-b54d-33112be20591" containerID="d182ca4ffd5f6689fd7eab051ff09863ca36bfa395816af241599dee3970a5d9" exitCode=0
Oct 31 03:42:51.133825 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.133738    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg" event={"ID":"017e3f11-748c-46c0-b54d-33112be20591","Type":"ContainerDied","Data":"d182ca4ffd5f6689fd7eab051ff09863ca36bfa395816af241599dee3970a5d9"}
Oct 31 03:42:51.133825 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.133802    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg" event={"ID":"017e3f11-748c-46c0-b54d-33112be20591","Type":"ContainerDied","Data":"282fd76f76265f2084b6bc66376fd8cca76c131633315fb0b3b613715e4ecf5d"}
Oct 31 03:42:51.134017 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.133827    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="282fd76f76265f2084b6bc66376fd8cca76c131633315fb0b3b613715e4ecf5d"
Oct 31 03:42:51.135743 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.135723    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:42:51.136236 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.136215    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-data\") pod \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") "
Oct 31 03:42:51.136330 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.136299    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-socket\") pod \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") "
Oct 31 03:42:51.136386 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.136373    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-envoy\") pod \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") "
Oct 31 03:42:51.136446 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.136413    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-certs\") pod \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") "
Oct 31 03:42:51.136504 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.136478    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istiod-ca-cert\") pod \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") "
Oct 31 03:42:51.136556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.136500    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-data" pod "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" (UID: "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:51.136556 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.136550    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-kube-api-access-ldprm\" (UniqueName: \"kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-kube-api-access-ldprm\") pod \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") "
Oct 31 03:42:51.137117 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.136590    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-token\" (UniqueName: \"kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-token\") pod \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") "
Oct 31 03:42:51.137117 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.136626    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-credential-socket\") pod \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") "
Oct 31 03:42:51.137117 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.136706    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-podinfo\") pod \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\" (UID: \"d2d8e6e8-60c1-48ef-9630-dc37ab279bf7\") "
Oct 31 03:42:51.137117 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.136588    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-socket" pod "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" (UID: "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:51.137117 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.136720    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-certs" pod "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" (UID: "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:51.137117 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.136933    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istiod-ca-cert" pod "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" (UID: "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:42:51.137376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.137131    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.137376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.137150    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.137376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.137166    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.137376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.137181    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.137376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.137241    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-credential-socket" pod "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" (UID: "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:51.138850 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.138804    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-envoy" pod "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" (UID: "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:51.139328 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.139307    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-kube-api-access-ldprm" pod "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" (UID: "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7"). InnerVolumeSpecName "kube-api-access-ldprm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:51.139698 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.139685    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-token" pod "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" (UID: "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:51.140002 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.139969    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-podinfo" pod "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" (UID: "d2d8e6e8-60c1-48ef-9630-dc37ab279bf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:42:51.143078 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.143056    5415 scope.go:122] "RemoveContainer" containerID="86cb37c0c6ca1ebb779f1c7d144e5531f613bea5d85f874ab89d48289413182f"
Oct 31 03:42:51.143417 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:42:51.143397    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86cb37c0c6ca1ebb779f1c7d144e5531f613bea5d85f874ab89d48289413182f\": container with ID starting with 86cb37c0c6ca1ebb779f1c7d144e5531f613bea5d85f874ab89d48289413182f not found: ID does not exist" containerID="86cb37c0c6ca1ebb779f1c7d144e5531f613bea5d85f874ab89d48289413182f"
Oct 31 03:42:51.143475 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.143424    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86cb37c0c6ca1ebb779f1c7d144e5531f613bea5d85f874ab89d48289413182f"} err="failed to get container status \"86cb37c0c6ca1ebb779f1c7d144e5531f613bea5d85f874ab89d48289413182f\": rpc error: code = NotFound desc = could not find container \"86cb37c0c6ca1ebb779f1c7d144e5531f613bea5d85f874ab89d48289413182f\": container with ID starting with 86cb37c0c6ca1ebb779f1c7d144e5531f613bea5d85f874ab89d48289413182f not found: ID does not exist"
Oct 31 03:42:51.237963 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.237923    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/017e3f11-748c-46c0-b54d-33112be20591-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017e3f11-748c-46c0-b54d-33112be20591-istiod-ca-cert\") pod \"017e3f11-748c-46c0-b54d-33112be20591\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") "
Oct 31 03:42:51.237963 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.237973    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/017e3f11-748c-46c0-b54d-33112be20591-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017e3f11-748c-46c0-b54d-33112be20591-istio-podinfo\") pod \"017e3f11-748c-46c0-b54d-33112be20591\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") "
Oct 31 03:42:51.238242 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.237998    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-socket\") pod \"017e3f11-748c-46c0-b54d-33112be20591\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") "
Oct 31 03:42:51.238242 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.238023    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-istio-token\" (UniqueName: \"kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-istio-token\") pod \"017e3f11-748c-46c0-b54d-33112be20591\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") "
Oct 31 03:42:51.238242 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.238139    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-data\") pod \"017e3f11-748c-46c0-b54d-33112be20591\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") "
Oct 31 03:42:51.238242 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.238199    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-credential-socket\") pod \"017e3f11-748c-46c0-b54d-33112be20591\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") "
Oct 31 03:42:51.238456 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.238247    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-certs\") pod \"017e3f11-748c-46c0-b54d-33112be20591\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") "
Oct 31 03:42:51.238456 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.238299    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-kube-api-access-88hhs\" (UniqueName: \"kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-kube-api-access-88hhs\") pod \"017e3f11-748c-46c0-b54d-33112be20591\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") "
Oct 31 03:42:51.238456 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.238339    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-envoy\") pod \"017e3f11-748c-46c0-b54d-33112be20591\" (UID: \"017e3f11-748c-46c0-b54d-33112be20591\") "
Oct 31 03:42:51.238456 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.238378    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-socket" pod "017e3f11-748c-46c0-b54d-33112be20591" (UID: "017e3f11-748c-46c0-b54d-33112be20591"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:51.238664 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.238462    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017e3f11-748c-46c0-b54d-33112be20591-istiod-ca-cert" pod "017e3f11-748c-46c0-b54d-33112be20591" (UID: "017e3f11-748c-46c0-b54d-33112be20591"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:42:51.238664 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.238522    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-certs" pod "017e3f11-748c-46c0-b54d-33112be20591" (UID: "017e3f11-748c-46c0-b54d-33112be20591"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:51.238664 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.238619    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-credential-socket" pod "017e3f11-748c-46c0-b54d-33112be20591" (UID: "017e3f11-748c-46c0-b54d-33112be20591"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:51.238790 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.238715    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-data" pod "017e3f11-748c-46c0-b54d-33112be20591" (UID: "017e3f11-748c-46c0-b54d-33112be20591"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:51.239016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.238967    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.239016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.238993    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017e3f11-748c-46c0-b54d-33112be20591-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.239016 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.239011    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.239250 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.239026    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.239250 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.239039    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.239250 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.239053    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.239250 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.239068    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.239250 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.239081    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.239250 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.239095    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldprm\" (UniqueName: \"kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-kube-api-access-ldprm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.239250 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.239109    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.240514 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.240488    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/017e3f11-748c-46c0-b54d-33112be20591-istio-podinfo" pod "017e3f11-748c-46c0-b54d-33112be20591" (UID: "017e3f11-748c-46c0-b54d-33112be20591"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:42:51.240626 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.240539    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-kube-api-access-88hhs" pod "017e3f11-748c-46c0-b54d-33112be20591" (UID: "017e3f11-748c-46c0-b54d-33112be20591"). InnerVolumeSpecName "kube-api-access-88hhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:51.240684 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.240643    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-envoy" pod "017e3f11-748c-46c0-b54d-33112be20591" (UID: "017e3f11-748c-46c0-b54d-33112be20591"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:51.240762 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.240740    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-istio-token" pod "017e3f11-748c-46c0-b54d-33112be20591" (UID: "017e3f11-748c-46c0-b54d-33112be20591"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:51.340037 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.339999    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88hhs\" (UniqueName: \"kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-kube-api-access-88hhs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.340037 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.340031    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017e3f11-748c-46c0-b54d-33112be20591-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.340250 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.340047    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017e3f11-748c-46c0-b54d-33112be20591-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.340250 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.340059    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/017e3f11-748c-46c0-b54d-33112be20591-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:51.548490 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.548453    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"]
Oct 31 03:42:51.554843 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:51.554815    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-cw5nj"]
Oct 31 03:42:52.141412 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:52.141374    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"
Oct 31 03:42:52.298078 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:52.298034    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"]
Oct 31 03:42:52.344661 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:52.344619    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-m2mtg"]
Oct 31 03:42:52.605946 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:52.605913    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="017e3f11-748c-46c0-b54d-33112be20591" path="/var/lib/kubelet/pods/017e3f11-748c-46c0-b54d-33112be20591/volumes"
Oct 31 03:42:52.606410 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:52.606395    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" path="/var/lib/kubelet/pods/d2d8e6e8-60c1-48ef-9630-dc37ab279bf7/volumes"
Oct 31 03:42:56.248245 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:56.248208    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"]
Oct 31 03:42:56.248733 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:42:56.248417    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerName="istio-proxy" containerID="cri-o://c9d909e102a83c70db3d8d24ad24acdfe02668d535841ccdb7dbc8e29f33b8ac" gracePeriod=30
Oct 31 03:43:01.509364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.509302    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:43:01.559975 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.559940    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-credential-socket\") pod \"d4743042-c4eb-407c-a8c1-6421259d5a3e\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") "
Oct 31 03:43:01.559975 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.559990    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-kube-api-access-zbgbr\" (UniqueName: \"kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-kube-api-access-zbgbr\") pod \"d4743042-c4eb-407c-a8c1-6421259d5a3e\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") "
Oct 31 03:43:01.560240 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560021    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-certs\") pod \"d4743042-c4eb-407c-a8c1-6421259d5a3e\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") "
Oct 31 03:43:01.560240 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560044    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-podinfo\") pod \"d4743042-c4eb-407c-a8c1-6421259d5a3e\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") "
Oct 31 03:43:01.560240 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560137    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-envoy\") pod \"d4743042-c4eb-407c-a8c1-6421259d5a3e\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") "
Oct 31 03:43:01.560240 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560173    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-socket\") pod \"d4743042-c4eb-407c-a8c1-6421259d5a3e\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") "
Oct 31 03:43:01.560240 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560214    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-token\" (UniqueName: \"kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-token\") pod \"d4743042-c4eb-407c-a8c1-6421259d5a3e\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") "
Oct 31 03:43:01.560494 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560254    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-data\") pod \"d4743042-c4eb-407c-a8c1-6421259d5a3e\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") "
Oct 31 03:43:01.560494 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560277    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-credential-socket" pod "d4743042-c4eb-407c-a8c1-6421259d5a3e" (UID: "d4743042-c4eb-407c-a8c1-6421259d5a3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:01.560494 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560303    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4743042-c4eb-407c-a8c1-6421259d5a3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4743042-c4eb-407c-a8c1-6421259d5a3e-istiod-ca-cert\") pod \"d4743042-c4eb-407c-a8c1-6421259d5a3e\" (UID: \"d4743042-c4eb-407c-a8c1-6421259d5a3e\") "
Oct 31 03:43:01.560654 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560480    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-socket" pod "d4743042-c4eb-407c-a8c1-6421259d5a3e" (UID: "d4743042-c4eb-407c-a8c1-6421259d5a3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:01.560654 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560551    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-data" pod "d4743042-c4eb-407c-a8c1-6421259d5a3e" (UID: "d4743042-c4eb-407c-a8c1-6421259d5a3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:01.560756 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560696    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-certs" pod "d4743042-c4eb-407c-a8c1-6421259d5a3e" (UID: "d4743042-c4eb-407c-a8c1-6421259d5a3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:01.560824 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560801    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.560824 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560811    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4743042-c4eb-407c-a8c1-6421259d5a3e-istiod-ca-cert" pod "d4743042-c4eb-407c-a8c1-6421259d5a3e" (UID: "d4743042-c4eb-407c-a8c1-6421259d5a3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:01.560966 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560830    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.560966 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560846    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.560966 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.560861    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.562474 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.562434    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-kube-api-access-zbgbr" pod "d4743042-c4eb-407c-a8c1-6421259d5a3e" (UID: "d4743042-c4eb-407c-a8c1-6421259d5a3e"). InnerVolumeSpecName "kube-api-access-zbgbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:01.562598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.562483    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-podinfo" pod "d4743042-c4eb-407c-a8c1-6421259d5a3e" (UID: "d4743042-c4eb-407c-a8c1-6421259d5a3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:43:01.562598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.562503    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-token" pod "d4743042-c4eb-407c-a8c1-6421259d5a3e" (UID: "d4743042-c4eb-407c-a8c1-6421259d5a3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:01.562598 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.562525    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-envoy" pod "d4743042-c4eb-407c-a8c1-6421259d5a3e" (UID: "d4743042-c4eb-407c-a8c1-6421259d5a3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:01.662028 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.661980    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.662028 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.662017    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.662028 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.662028    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4743042-c4eb-407c-a8c1-6421259d5a3e-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.662028 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.662038    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbgbr\" (UniqueName: \"kubernetes.io/projected/d4743042-c4eb-407c-a8c1-6421259d5a3e-kube-api-access-zbgbr\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.662346 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:01.662049    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4743042-c4eb-407c-a8c1-6421259d5a3e-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:02.200302 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:02.200265    5415 generic.go:356] "Generic (PLEG): container finished" podID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerID="c9d909e102a83c70db3d8d24ad24acdfe02668d535841ccdb7dbc8e29f33b8ac" exitCode=0
Oct 31 03:43:02.200520 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:02.200334    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"
Oct 31 03:43:02.200520 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:02.200343    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" event={"ID":"d4743042-c4eb-407c-a8c1-6421259d5a3e","Type":"ContainerDied","Data":"c9d909e102a83c70db3d8d24ad24acdfe02668d535841ccdb7dbc8e29f33b8ac"}
Oct 31 03:43:02.200520 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:02.200450    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2" event={"ID":"d4743042-c4eb-407c-a8c1-6421259d5a3e","Type":"ContainerDied","Data":"5a8cedd59c00b3e8340967228527d571f83da411b037b7699ff5fc2e7812a2a9"}
Oct 31 03:43:02.200520 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:02.200483    5415 scope.go:122] "RemoveContainer" containerID="c9d909e102a83c70db3d8d24ad24acdfe02668d535841ccdb7dbc8e29f33b8ac"
Oct 31 03:43:02.212308 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:02.212285    5415 scope.go:122] "RemoveContainer" containerID="c9d909e102a83c70db3d8d24ad24acdfe02668d535841ccdb7dbc8e29f33b8ac"
Oct 31 03:43:02.212642 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:43:02.212615    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d909e102a83c70db3d8d24ad24acdfe02668d535841ccdb7dbc8e29f33b8ac\": container with ID starting with c9d909e102a83c70db3d8d24ad24acdfe02668d535841ccdb7dbc8e29f33b8ac not found: ID does not exist" containerID="c9d909e102a83c70db3d8d24ad24acdfe02668d535841ccdb7dbc8e29f33b8ac"
Oct 31 03:43:02.212722 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:02.212655    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d909e102a83c70db3d8d24ad24acdfe02668d535841ccdb7dbc8e29f33b8ac"} err="failed to get container status \"c9d909e102a83c70db3d8d24ad24acdfe02668d535841ccdb7dbc8e29f33b8ac\": rpc error: code = NotFound desc = could not find container \"c9d909e102a83c70db3d8d24ad24acdfe02668d535841ccdb7dbc8e29f33b8ac\": container with ID starting with c9d909e102a83c70db3d8d24ad24acdfe02668d535841ccdb7dbc8e29f33b8ac not found: ID does not exist"
Oct 31 03:43:02.404575 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:02.404541    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"]
Oct 31 03:43:02.451911 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:02.451807    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-mwcs2"]
Oct 31 03:43:02.607517 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:02.607485    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" path="/var/lib/kubelet/pods/d4743042-c4eb-407c-a8c1-6421259d5a3e/volumes"
Oct 31 03:43:11.582506 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.582465    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"]
Oct 31 03:43:11.583062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.582976    5415 cpu_manager.go:414] "containerMap: removing container" podUID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" containerName="istio-proxy"
Oct 31 03:43:11.583062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.582987    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" containerName="istio-proxy"
Oct 31 03:43:11.583062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.583003    5415 cpu_manager.go:414] "containerMap: removing container" podUID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" containerName="istio-proxy"
Oct 31 03:43:11.583062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.583007    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" containerName="istio-proxy"
Oct 31 03:43:11.583062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.583024    5415 cpu_manager.go:414] "containerMap: removing container" podUID="017e3f11-748c-46c0-b54d-33112be20591" containerName="istio-proxy"
Oct 31 03:43:11.583062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.583030    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="017e3f11-748c-46c0-b54d-33112be20591" containerName="istio-proxy"
Oct 31 03:43:11.583062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.583042    5415 cpu_manager.go:414] "containerMap: removing container" podUID="1b7cef69-f9b2-407d-a283-4f292a19b91b" containerName="payload-pre-processing"
Oct 31 03:43:11.583062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.583046    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7cef69-f9b2-407d-a283-4f292a19b91b" containerName="payload-pre-processing"
Oct 31 03:43:11.583062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.583058    5415 cpu_manager.go:414] "containerMap: removing container" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerName="istio-proxy"
Oct 31 03:43:11.583062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.583063    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerName="istio-proxy"
Oct 31 03:43:11.583380 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.583141    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ec02cc-eaaa-4913-8904-6a97d51a0d46" containerName="istio-proxy"
Oct 31 03:43:11.583380 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.583150    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d8e6e8-60c1-48ef-9630-dc37ab279bf7" containerName="istio-proxy"
Oct 31 03:43:11.583380 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.583157    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="017e3f11-748c-46c0-b54d-33112be20591" containerName="istio-proxy"
Oct 31 03:43:11.583380 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.583163    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7cef69-f9b2-407d-a283-4f292a19b91b" containerName="payload-pre-processing"
Oct 31 03:43:11.583380 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.583169    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4743042-c4eb-407c-a8c1-6421259d5a3e" containerName="istio-proxy"
Oct 31 03:43:11.588272 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.588250    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"
Oct 31 03:43:11.643712 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.643669    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"]
Oct 31 03:43:11.674251 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.674200    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52hcb\" (UniqueName: \"kubernetes.io/projected/da356000-9c44-4fa1-9d31-0dd6e65145f4-kube-api-access-52hcb\") pod \"payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch\" (UID: \"da356000-9c44-4fa1-9d31-0dd6e65145f4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"
Oct 31 03:43:11.674571 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.674544    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da356000-9c44-4fa1-9d31-0dd6e65145f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch\" (UID: \"da356000-9c44-4fa1-9d31-0dd6e65145f4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"
Oct 31 03:43:11.729275 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.729240    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-03d737-dockercfg-bptxj\""
Oct 31 03:43:11.732956 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.732914    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-03d737\""
Oct 31 03:43:11.743944 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.743912    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"]
Oct 31 03:43:11.747986 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.747964    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"
Oct 31 03:43:11.775856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.775819    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da356000-9c44-4fa1-9d31-0dd6e65145f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch\" (UID: \"da356000-9c44-4fa1-9d31-0dd6e65145f4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"
Oct 31 03:43:11.776062 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.775871    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52hcb\" (UniqueName: \"kubernetes.io/projected/da356000-9c44-4fa1-9d31-0dd6e65145f4-kube-api-access-52hcb\") pod \"payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch\" (UID: \"da356000-9c44-4fa1-9d31-0dd6e65145f4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"
Oct 31 03:43:11.776518 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.776498    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da356000-9c44-4fa1-9d31-0dd6e65145f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch\" (UID: \"da356000-9c44-4fa1-9d31-0dd6e65145f4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"
Oct 31 03:43:11.780861 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.780833    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"]
Oct 31 03:43:11.829854 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.829822    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52hcb\" (UniqueName: \"kubernetes.io/projected/da356000-9c44-4fa1-9d31-0dd6e65145f4-kube-api-access-52hcb\") pod \"payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch\" (UID: \"da356000-9c44-4fa1-9d31-0dd6e65145f4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"
Oct 31 03:43:11.876857 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.876762    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c780d3a-6662-4c40-b816-3cf1f6da41ff-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x\" (UID: \"8c780d3a-6662-4c40-b816-3cf1f6da41ff\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"
Oct 31 03:43:11.877050 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.876987    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk7qc\" (UniqueName: \"kubernetes.io/projected/8c780d3a-6662-4c40-b816-3cf1f6da41ff-kube-api-access-rk7qc\") pod \"payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x\" (UID: \"8c780d3a-6662-4c40-b816-3cf1f6da41ff\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"
Oct 31 03:43:11.901484 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.901450    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"
Oct 31 03:43:11.977821 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.977782    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk7qc\" (UniqueName: \"kubernetes.io/projected/8c780d3a-6662-4c40-b816-3cf1f6da41ff-kube-api-access-rk7qc\") pod \"payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x\" (UID: \"8c780d3a-6662-4c40-b816-3cf1f6da41ff\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"
Oct 31 03:43:11.978026 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.977917    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c780d3a-6662-4c40-b816-3cf1f6da41ff-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x\" (UID: \"8c780d3a-6662-4c40-b816-3cf1f6da41ff\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"
Oct 31 03:43:11.978908 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:11.978865    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c780d3a-6662-4c40-b816-3cf1f6da41ff-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x\" (UID: \"8c780d3a-6662-4c40-b816-3cf1f6da41ff\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"
Oct 31 03:43:12.032225 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:12.032195    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk7qc\" (UniqueName: \"kubernetes.io/projected/8c780d3a-6662-4c40-b816-3cf1f6da41ff-kube-api-access-rk7qc\") pod \"payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x\" (UID: \"8c780d3a-6662-4c40-b816-3cf1f6da41ff\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"
Oct 31 03:43:12.061102 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:12.061062    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"
Oct 31 03:43:12.238853 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:12.238768    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"]
Oct 31 03:43:12.270515 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:12.270482    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"]
Oct 31 03:43:12.272523 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:43:12.272492    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda356000_9c44_4fa1_9d31_0dd6e65145f4.slice/crio-d291bfb3784d3a2d957103395daebe820184347fd70d3d4df0e0bee6571a9290 WatchSource:0}: Error finding container d291bfb3784d3a2d957103395daebe820184347fd70d3d4df0e0bee6571a9290: Status 404 returned error can't find the container with id d291bfb3784d3a2d957103395daebe820184347fd70d3d4df0e0bee6571a9290
Oct 31 03:43:12.342292 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:12.342262    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"]
Oct 31 03:43:12.428068 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:12.428044    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"]
Oct 31 03:43:13.275544 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:13.275507    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x" event={"ID":"8c780d3a-6662-4c40-b816-3cf1f6da41ff","Type":"ContainerStarted","Data":"dd07e4a1b44467e8c3ae58f85c84e6262da6a18dd167c55f2e4890e8c97f313b"}
Oct 31 03:43:13.276098 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:13.275568    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x" event={"ID":"8c780d3a-6662-4c40-b816-3cf1f6da41ff","Type":"ContainerStarted","Data":"591862ffe2d5b805208cbcdbfca10f456f724cc9ce16a8d857d4ff44578e3122"}
Oct 31 03:43:13.276098 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:13.275663    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"
Oct 31 03:43:13.277084 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:13.277057    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch" event={"ID":"da356000-9c44-4fa1-9d31-0dd6e65145f4","Type":"ContainerStarted","Data":"ed655209e105418907f381a8204d2cc81fc8fd27d3c87736c26a192fb022dfca"}
Oct 31 03:43:13.277214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:13.277095    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch" event={"ID":"da356000-9c44-4fa1-9d31-0dd6e65145f4","Type":"ContainerStarted","Data":"d291bfb3784d3a2d957103395daebe820184347fd70d3d4df0e0bee6571a9290"}
Oct 31 03:43:13.277214 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:13.277177    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"
Oct 31 03:43:13.384245 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:13.384193    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x" podStartSLOduration=2.384175498 podStartE2EDuration="2.384175498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:13.38234853 +0000 UTC m=+2134.942828283" watchObservedRunningTime="2026-10-31 03:43:13.384175498 +0000 UTC m=+2134.944655233"
Oct 31 03:43:13.491509 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:13.491457    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch" podStartSLOduration=2.491442612 podStartE2EDuration="2.491442612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:13.48986805 +0000 UTC m=+2135.050347782" watchObservedRunningTime="2026-10-31 03:43:13.491442612 +0000 UTC m=+2135.051922374"
Oct 31 03:43:16.176102 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:43:16.176064    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:43:17.239389 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.239343    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj" podUID="bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a" containerName="payload-processing" containerID="cri-o://5ac545eb5eeed9d76504067f4a2a68a3c47d534c7ff0964d2f082544d03614d6" gracePeriod=25
Oct 31 03:43:17.342653 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.342606    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr" podUID="0e9f51f4-cb08-4e82-b8d4-00694552fd27" containerName="payload-pre-processing" containerID="cri-o://d4f85c053ec7f599f52a09a525b63d0037d3752cf31b586f5173b6c7530c011b" gracePeriod=25
Oct 31 03:43:17.542533 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.542507    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"
Oct 31 03:43:17.648424 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.648391    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-plugins-config-volume\") pod \"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a\" (UID: \"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a\") "
Oct 31 03:43:17.648636 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.648619    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-kube-api-access-2h76q\" (UniqueName: \"kubernetes.io/projected/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-kube-api-access-2h76q\") pod \"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a\" (UID: \"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a\") "
Oct 31 03:43:17.648816 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.648790    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-plugins-config-volume" pod "bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a" (UID: "bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:17.649092 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.649068    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:17.651215 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.651134    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-kube-api-access-2h76q" pod "bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a" (UID: "bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a"). InnerVolumeSpecName "kube-api-access-2h76q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:17.661051 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.661023    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"
Oct 31 03:43:17.739941 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.739871    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"]
Oct 31 03:43:17.740398 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.740385    5415 cpu_manager.go:414] "containerMap: removing container" podUID="bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a" containerName="payload-processing"
Oct 31 03:43:17.740445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.740399    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a" containerName="payload-processing"
Oct 31 03:43:17.740445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.740428    5415 cpu_manager.go:414] "containerMap: removing container" podUID="0e9f51f4-cb08-4e82-b8d4-00694552fd27" containerName="payload-pre-processing"
Oct 31 03:43:17.740445 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.740433    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9f51f4-cb08-4e82-b8d4-00694552fd27" containerName="payload-pre-processing"
Oct 31 03:43:17.740536 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.740509    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a" containerName="payload-processing"
Oct 31 03:43:17.740536 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.740520    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9f51f4-cb08-4e82-b8d4-00694552fd27" containerName="payload-pre-processing"
Oct 31 03:43:17.743911 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.743863    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.750189 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.750097    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9f51f4-cb08-4e82-b8d4-00694552fd27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9f51f4-cb08-4e82-b8d4-00694552fd27-plugins-config-volume\") pod \"0e9f51f4-cb08-4e82-b8d4-00694552fd27\" (UID: \"0e9f51f4-cb08-4e82-b8d4-00694552fd27\") "
Oct 31 03:43:17.750323 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.750228    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9f51f4-cb08-4e82-b8d4-00694552fd27-kube-api-access-nfdkx\" (UniqueName: \"kubernetes.io/projected/0e9f51f4-cb08-4e82-b8d4-00694552fd27-kube-api-access-nfdkx\") pod \"0e9f51f4-cb08-4e82-b8d4-00694552fd27\" (UID: \"0e9f51f4-cb08-4e82-b8d4-00694552fd27\") "
Oct 31 03:43:17.750466 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.750445    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9f51f4-cb08-4e82-b8d4-00694552fd27-plugins-config-volume" pod "0e9f51f4-cb08-4e82-b8d4-00694552fd27" (UID: "0e9f51f4-cb08-4e82-b8d4-00694552fd27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:17.750674 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.750651    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9f51f4-cb08-4e82-b8d4-00694552fd27-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:17.750674 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.750675    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h76q\" (UniqueName: \"kubernetes.io/projected/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a-kube-api-access-2h76q\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:17.752538 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.752518    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9f51f4-cb08-4e82-b8d4-00694552fd27-kube-api-access-nfdkx" pod "0e9f51f4-cb08-4e82-b8d4-00694552fd27" (UID: "0e9f51f4-cb08-4e82-b8d4-00694552fd27"). InnerVolumeSpecName "kube-api-access-nfdkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:17.793500 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.793472    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a32cdb\"/\"kube-root-ca.crt\""
Oct 31 03:43:17.793672 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.793562    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a32cdb\"/\"ipp-route-a32cdb-kserve-self-signed-certs\""
Oct 31 03:43:17.793672 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.793636    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a32cdb\"/\"default-dockercfg-c7qnk\""
Oct 31 03:43:17.800463 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.800433    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"]
Oct 31 03:43:17.851824 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.851782    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-home\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.852033 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.851972    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-dshm\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.852109 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.852090    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-model-cache\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.852167 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.852120    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87788dd8-2aa4-4b65-80f2-8116be88a903-tls-certs\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.852208 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.852184    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97wh\" (UniqueName: \"kubernetes.io/projected/87788dd8-2aa4-4b65-80f2-8116be88a903-kube-api-access-d97wh\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.852250 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.852219    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-tmp-dir\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.852328 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.852313    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfdkx\" (UniqueName: \"kubernetes.io/projected/0e9f51f4-cb08-4e82-b8d4-00694552fd27-kube-api-access-nfdkx\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:17.902486 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.902455    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a32cdb\"/\"openshift-service-ca.crt\""
Oct 31 03:43:17.953242 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.953200    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-tmp-dir\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.953462 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.953308    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-home\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.953462 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.953440    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-dshm\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.953585 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.953481    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-model-cache\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.953585 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.953507    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87788dd8-2aa4-4b65-80f2-8116be88a903-tls-certs\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.953585 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.953540    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97wh\" (UniqueName: \"kubernetes.io/projected/87788dd8-2aa4-4b65-80f2-8116be88a903-kube-api-access-d97wh\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.953729 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.953604    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-tmp-dir\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.953729 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.953680    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-home\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.953824 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.953768    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-model-cache\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.955810 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.955781    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-dshm\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:17.956073 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:17.956057    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87788dd8-2aa4-4b65-80f2-8116be88a903-tls-certs\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:18.011031 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.010938    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97wh\" (UniqueName: \"kubernetes.io/projected/87788dd8-2aa4-4b65-80f2-8116be88a903-kube-api-access-d97wh\") pod \"ipp-route-a32cdb-kserve-cfbc9f797-snmsp\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") " pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:18.056997 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.056960    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:18.306727 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.306690    5415 generic.go:356] "Generic (PLEG): container finished" podID="0e9f51f4-cb08-4e82-b8d4-00694552fd27" containerID="d4f85c053ec7f599f52a09a525b63d0037d3752cf31b586f5173b6c7530c011b" exitCode=0
Oct 31 03:43:18.307255 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.306741    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"
Oct 31 03:43:18.307255 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.306771    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr" event={"ID":"0e9f51f4-cb08-4e82-b8d4-00694552fd27","Type":"ContainerDied","Data":"d4f85c053ec7f599f52a09a525b63d0037d3752cf31b586f5173b6c7530c011b"}
Oct 31 03:43:18.307255 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.306824    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr" event={"ID":"0e9f51f4-cb08-4e82-b8d4-00694552fd27","Type":"ContainerDied","Data":"28d6c5c6c1c9f3e0bbcfde401bf6a5c5d92773ca649fb2a49ea60f9fe63b7d6d"}
Oct 31 03:43:18.307255 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.306846    5415 scope.go:122] "RemoveContainer" containerID="d4f85c053ec7f599f52a09a525b63d0037d3752cf31b586f5173b6c7530c011b"
Oct 31 03:43:18.309091 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.309065    5415 generic.go:356] "Generic (PLEG): container finished" podID="bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a" containerID="5ac545eb5eeed9d76504067f4a2a68a3c47d534c7ff0964d2f082544d03614d6" exitCode=0
Oct 31 03:43:18.309229 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.309128    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"
Oct 31 03:43:18.309229 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.309202    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj" event={"ID":"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a","Type":"ContainerDied","Data":"5ac545eb5eeed9d76504067f4a2a68a3c47d534c7ff0964d2f082544d03614d6"}
Oct 31 03:43:18.309629 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.309277    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj" event={"ID":"bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a","Type":"ContainerDied","Data":"a3daaf708802585419f56bde932e5a7b6e119fbc282534b8c15419b9d00c7f5a"}
Oct 31 03:43:18.319237 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.319213    5415 scope.go:122] "RemoveContainer" containerID="d4f85c053ec7f599f52a09a525b63d0037d3752cf31b586f5173b6c7530c011b"
Oct 31 03:43:18.319611 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:43:18.319583    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f85c053ec7f599f52a09a525b63d0037d3752cf31b586f5173b6c7530c011b\": container with ID starting with d4f85c053ec7f599f52a09a525b63d0037d3752cf31b586f5173b6c7530c011b not found: ID does not exist" containerID="d4f85c053ec7f599f52a09a525b63d0037d3752cf31b586f5173b6c7530c011b"
Oct 31 03:43:18.319719 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.319620    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f85c053ec7f599f52a09a525b63d0037d3752cf31b586f5173b6c7530c011b"} err="failed to get container status \"d4f85c053ec7f599f52a09a525b63d0037d3752cf31b586f5173b6c7530c011b\": rpc error: code = NotFound desc = could not find container \"d4f85c053ec7f599f52a09a525b63d0037d3752cf31b586f5173b6c7530c011b\": container with ID starting with d4f85c053ec7f599f52a09a525b63d0037d3752cf31b586f5173b6c7530c011b not found: ID does not exist"
Oct 31 03:43:18.319719 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.319643    5415 scope.go:122] "RemoveContainer" containerID="5ac545eb5eeed9d76504067f4a2a68a3c47d534c7ff0964d2f082544d03614d6"
Oct 31 03:43:18.329388 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.329363    5415 scope.go:122] "RemoveContainer" containerID="5ac545eb5eeed9d76504067f4a2a68a3c47d534c7ff0964d2f082544d03614d6"
Oct 31 03:43:18.329777 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:43:18.329746    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac545eb5eeed9d76504067f4a2a68a3c47d534c7ff0964d2f082544d03614d6\": container with ID starting with 5ac545eb5eeed9d76504067f4a2a68a3c47d534c7ff0964d2f082544d03614d6 not found: ID does not exist" containerID="5ac545eb5eeed9d76504067f4a2a68a3c47d534c7ff0964d2f082544d03614d6"
Oct 31 03:43:18.329910 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.329788    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac545eb5eeed9d76504067f4a2a68a3c47d534c7ff0964d2f082544d03614d6"} err="failed to get container status \"5ac545eb5eeed9d76504067f4a2a68a3c47d534c7ff0964d2f082544d03614d6\": rpc error: code = NotFound desc = could not find container \"5ac545eb5eeed9d76504067f4a2a68a3c47d534c7ff0964d2f082544d03614d6\": container with ID starting with 5ac545eb5eeed9d76504067f4a2a68a3c47d534c7ff0964d2f082544d03614d6 not found: ID does not exist"
Oct 31 03:43:18.458469 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.458444    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"]
Oct 31 03:43:18.461524 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:43:18.461495    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87788dd8_2aa4_4b65_80f2_8116be88a903.slice/crio-bef7f2dcac8752e69029ba88ec54a9fd9fdd4fc1f7d4e32219ba6c23abbf0975 WatchSource:0}: Error finding container bef7f2dcac8752e69029ba88ec54a9fd9fdd4fc1f7d4e32219ba6c23abbf0975: Status 404 returned error can't find the container with id bef7f2dcac8752e69029ba88ec54a9fd9fdd4fc1f7d4e32219ba6c23abbf0975
Oct 31 03:43:18.463646 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.463623    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"]
Oct 31 03:43:18.467948 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.467922    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c561e3-645dc66449-jzksj"]
Oct 31 03:43:18.574729 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.574694    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"]
Oct 31 03:43:18.577746 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.577715    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-779c466475-q8hdr"]
Oct 31 03:43:18.606218 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.606185    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9f51f4-cb08-4e82-b8d4-00694552fd27" path="/var/lib/kubelet/pods/0e9f51f4-cb08-4e82-b8d4-00694552fd27/volumes"
Oct 31 03:43:18.606599 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:18.606586    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a" path="/var/lib/kubelet/pods/bd6d4ed8-fc33-407d-82ee-03cd5f4cb22a/volumes"
Oct 31 03:43:19.316033 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:19.315984    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp" event={"ID":"87788dd8-2aa4-4b65-80f2-8116be88a903","Type":"ContainerStarted","Data":"a14fa56e76e1d20e441fef56c8e77acc98089feb60b039d6ef5d093f6d44ca5e"}
Oct 31 03:43:19.316468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:19.316043    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp" event={"ID":"87788dd8-2aa4-4b65-80f2-8116be88a903","Type":"ContainerStarted","Data":"bef7f2dcac8752e69029ba88ec54a9fd9fdd4fc1f7d4e32219ba6c23abbf0975"}
Oct 31 03:43:19.426327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:19.426274    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp" podStartSLOduration=2.426258466 podStartE2EDuration="2.426258466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:19.425154462 +0000 UTC m=+2140.985634193" watchObservedRunningTime="2026-10-31 03:43:19.426258466 +0000 UTC m=+2140.986738196"
Oct 31 03:43:20.057778 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:20.057735    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:24.286722 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:24.286689    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"
Oct 31 03:43:24.287150 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:24.286771    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"
Oct 31 03:43:28.057464 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:28.057416    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:28.070198 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:28.070169    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:28.389504 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:28.389467    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:43:29.023603 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:29.023563    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"]
Oct 31 03:43:29.027357 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:29.027334    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"
Oct 31 03:43:29.090216 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:29.090180    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"]
Oct 31 03:43:29.097969 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:29.097926    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8fq\" (UniqueName: \"kubernetes.io/projected/3dd144db-5910-4f9f-bec3-3655053c0152-kube-api-access-lm8fq\") pod \"payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf\" (UID: \"3dd144db-5910-4f9f-bec3-3655053c0152\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"
Oct 31 03:43:29.098142 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:29.098103    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd144db-5910-4f9f-bec3-3655053c0152-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf\" (UID: \"3dd144db-5910-4f9f-bec3-3655053c0152\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"
Oct 31 03:43:29.131337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:29.131300    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-dab9ee-dockercfg-h5bzs\""
Oct 31 03:43:29.131337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:29.131300    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-dab9ee\""
Oct 31 03:43:29.199227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:29.199188    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd144db-5910-4f9f-bec3-3655053c0152-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf\" (UID: \"3dd144db-5910-4f9f-bec3-3655053c0152\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"
Oct 31 03:43:29.199426 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:29.199332    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8fq\" (UniqueName: \"kubernetes.io/projected/3dd144db-5910-4f9f-bec3-3655053c0152-kube-api-access-lm8fq\") pod \"payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf\" (UID: \"3dd144db-5910-4f9f-bec3-3655053c0152\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"
Oct 31 03:43:29.199921 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:29.199899    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd144db-5910-4f9f-bec3-3655053c0152-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf\" (UID: \"3dd144db-5910-4f9f-bec3-3655053c0152\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"
Oct 31 03:43:29.253808 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:29.253769    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8fq\" (UniqueName: \"kubernetes.io/projected/3dd144db-5910-4f9f-bec3-3655053c0152-kube-api-access-lm8fq\") pod \"payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf\" (UID: \"3dd144db-5910-4f9f-bec3-3655053c0152\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"
Oct 31 03:43:29.341791 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:29.341756    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"
Oct 31 03:43:29.715871 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:29.715839    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"]
Oct 31 03:43:29.718241 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:43:29.718206    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd144db_5910_4f9f_bec3_3655053c0152.slice/crio-6fa99d7bc8d05e706a85df5dea5301a081ccffd0445dc4ff741929bf3a3805a5 WatchSource:0}: Error finding container 6fa99d7bc8d05e706a85df5dea5301a081ccffd0445dc4ff741929bf3a3805a5: Status 404 returned error can't find the container with id 6fa99d7bc8d05e706a85df5dea5301a081ccffd0445dc4ff741929bf3a3805a5
Oct 31 03:43:30.392522 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:30.392481    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf" event={"ID":"3dd144db-5910-4f9f-bec3-3655053c0152","Type":"ContainerStarted","Data":"6b288065eb113604026b2eb4e4d56d549adad300501a45246ee1d18511d21d70"}
Oct 31 03:43:30.392972 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:30.392537    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf" event={"ID":"3dd144db-5910-4f9f-bec3-3655053c0152","Type":"ContainerStarted","Data":"6fa99d7bc8d05e706a85df5dea5301a081ccffd0445dc4ff741929bf3a3805a5"}
Oct 31 03:43:30.392972 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:30.392593    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"
Oct 31 03:43:30.506345 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:30.506293    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf" podStartSLOduration=2.5062762640000003 podStartE2EDuration="2.506276264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:30.505828974 +0000 UTC m=+2152.066308727" watchObservedRunningTime="2026-10-31 03:43:30.506276264 +0000 UTC m=+2152.066755995"
Oct 31 03:43:34.316216 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:34.316174    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"]
Oct 31 03:43:34.321689 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:34.321657    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"
Oct 31 03:43:34.362065 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:34.362030    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdxnf\" (UniqueName: \"kubernetes.io/projected/b529469f-b83f-41af-a9a3-35f55b1eab48-kube-api-access-wdxnf\") pod \"payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42\" (UID: \"b529469f-b83f-41af-a9a3-35f55b1eab48\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"
Oct 31 03:43:34.362288 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:34.362096    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b529469f-b83f-41af-a9a3-35f55b1eab48-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42\" (UID: \"b529469f-b83f-41af-a9a3-35f55b1eab48\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"
Oct 31 03:43:34.385728 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:34.385692    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"]
Oct 31 03:43:34.427954 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:34.427926    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d22078\""
Oct 31 03:43:34.428138 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:34.427957    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d22078-dockercfg-hfrkt\""
Oct 31 03:43:34.463628 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:34.463588    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdxnf\" (UniqueName: \"kubernetes.io/projected/b529469f-b83f-41af-a9a3-35f55b1eab48-kube-api-access-wdxnf\") pod \"payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42\" (UID: \"b529469f-b83f-41af-a9a3-35f55b1eab48\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"
Oct 31 03:43:34.463803 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:34.463681    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b529469f-b83f-41af-a9a3-35f55b1eab48-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42\" (UID: \"b529469f-b83f-41af-a9a3-35f55b1eab48\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"
Oct 31 03:43:34.464444 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:34.464420    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b529469f-b83f-41af-a9a3-35f55b1eab48-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42\" (UID: \"b529469f-b83f-41af-a9a3-35f55b1eab48\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"
Oct 31 03:43:34.525270 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:34.525233    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdxnf\" (UniqueName: \"kubernetes.io/projected/b529469f-b83f-41af-a9a3-35f55b1eab48-kube-api-access-wdxnf\") pod \"payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42\" (UID: \"b529469f-b83f-41af-a9a3-35f55b1eab48\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"
Oct 31 03:43:34.650330 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:34.650294    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"
Oct 31 03:43:35.031594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:35.031505    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"]
Oct 31 03:43:35.034494 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:43:35.034463    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb529469f_b83f_41af_a9a3_35f55b1eab48.slice/crio-6090aa152af40d92c17399b0c44f71e22f8e63d83b142b723b7d4c7c2388d205 WatchSource:0}: Error finding container 6090aa152af40d92c17399b0c44f71e22f8e63d83b142b723b7d4c7c2388d205: Status 404 returned error can't find the container with id 6090aa152af40d92c17399b0c44f71e22f8e63d83b142b723b7d4c7c2388d205
Oct 31 03:43:35.429091 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:35.429047    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42" event={"ID":"b529469f-b83f-41af-a9a3-35f55b1eab48","Type":"ContainerStarted","Data":"f7404e3d1aee147c2a82dfda365c5181c2ed29bba12b702b8e69a2414e183e01"}
Oct 31 03:43:35.429577 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:35.429112    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"
Oct 31 03:43:35.429577 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:35.429135    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42" event={"ID":"b529469f-b83f-41af-a9a3-35f55b1eab48","Type":"ContainerStarted","Data":"6090aa152af40d92c17399b0c44f71e22f8e63d83b142b723b7d4c7c2388d205"}
Oct 31 03:43:35.539215 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:35.539157    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42" podStartSLOduration=1.539140933 podStartE2EDuration="1.539140933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:35.538136525 +0000 UTC m=+2157.098616258" watchObservedRunningTime="2026-10-31 03:43:35.539140933 +0000 UTC m=+2157.099620663"
Oct 31 03:43:39.093314 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:39.093281    5415 scope.go:122] "RemoveContainer" containerID="d182ca4ffd5f6689fd7eab051ff09863ca36bfa395816af241599dee3970a5d9"
Oct 31 03:43:41.402567 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:41.402535    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"
Oct 31 03:43:46.175322 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:43:46.175242    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:43:46.438714 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:43:46.438631    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"
Oct 31 03:44:03.544151 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.544113    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"]
Oct 31 03:44:03.548166 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.548143    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:03.596621 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.596582    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:03.596796 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.596739    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg4vf\" (UniqueName: \"kubernetes.io/projected/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-kube-api-access-mg4vf\") pod \"maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:03.596796 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.596778    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-metrics-certs\") pod \"maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:03.596905 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.596809    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:03.597468 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.597449    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-03d737-serving-cert\""
Oct 31 03:44:03.605147 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.605114    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"]
Oct 31 03:44:03.697660 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.697624    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-metrics-certs\") pod \"maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:03.697890 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.697685    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:03.697890 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.697731    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:03.698019 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.697923    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg4vf\" (UniqueName: \"kubernetes.io/projected/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-kube-api-access-mg4vf\") pod \"maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:03.698173 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:44:03.698153    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-serving-cert: secret "maas-api-e2e-praxis-b-03d737-serving-cert" not found
Oct 31 03:44:03.698339 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:44:03.698324    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-maas-api-tls podName:f58c5c50-b168-4fc0-9ea1-6fc093b2007e nodeName:}" failed. No retries permitted until 2026-10-31 03:44:04.198304775 +0000 UTC m=+2185.758784489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-maas-api-tls") pod "maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4" (UID: "f58c5c50-b168-4fc0-9ea1-6fc093b2007e") : secret "maas-api-e2e-praxis-b-03d737-serving-cert" not found
Oct 31 03:44:03.698953 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.698926    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:03.701024 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.700998    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-metrics-certs\") pod \"maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:03.752321 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:03.752289    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg4vf\" (UniqueName: \"kubernetes.io/projected/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-kube-api-access-mg4vf\") pod \"maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:04.204829 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:04.204794    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:04.207389 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:04.207364    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:04.461743 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:04.461657    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:04.831227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:04.831144    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"]
Oct 31 03:44:04.834886 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:44:04.834850    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58c5c50_b168_4fc0_9ea1_6fc093b2007e.slice/crio-c76753ec8689a27aa2b81522817a30173c26d5f995d0f909c1e34a5a1ebcbb24 WatchSource:0}: Error finding container c76753ec8689a27aa2b81522817a30173c26d5f995d0f909c1e34a5a1ebcbb24: Status 404 returned error can't find the container with id c76753ec8689a27aa2b81522817a30173c26d5f995d0f909c1e34a5a1ebcbb24
Oct 31 03:44:04.836529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:04.836508    5415 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:44:05.604970 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:05.604935    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4" event={"ID":"f58c5c50-b168-4fc0-9ea1-6fc093b2007e","Type":"ContainerStarted","Data":"c76753ec8689a27aa2b81522817a30173c26d5f995d0f909c1e34a5a1ebcbb24"}
Oct 31 03:44:06.611258 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:06.611221    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4" event={"ID":"f58c5c50-b168-4fc0-9ea1-6fc093b2007e","Type":"ContainerStarted","Data":"388f8669c98b3a4bfcbd7fbf6893af4dda3897cbfbfacdedda598c310089b687"}
Oct 31 03:44:06.611678 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:06.611315    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:06.722265 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:06.722206    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4" podStartSLOduration=2.645264763 podStartE2EDuration="3.722188473s" totalImagesPullingTime="1.07692371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:06.719535389 +0000 UTC m=+2188.280015131" watchObservedRunningTime="2026-10-31 03:44:06.722188473 +0000 UTC m=+2188.282668203"
Oct 31 03:44:12.623096 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:12.623067    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:44:16.176603 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:44:16.176563    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:44:33.047945 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.047894    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"]
Oct 31 03:44:33.051966 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.051944    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.101327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.101290    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-d22078-serving-cert\""
Oct 31 03:44:33.139665 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.139616    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97qn5\" (UniqueName: \"kubernetes.io/projected/86d05e78-8832-4213-b976-f7b5b42c0863-kube-api-access-97qn5\") pod \"maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.139864 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.139844    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-metrics-certs\") pod \"maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.139943 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.139905    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.140060 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.139977    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d05e78-8832-4213-b976-f7b5b42c0863-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.150499 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.150468    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"]
Oct 31 03:44:33.241446 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.241402    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qn5\" (UniqueName: \"kubernetes.io/projected/86d05e78-8832-4213-b976-f7b5b42c0863-kube-api-access-97qn5\") pod \"maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.241642 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.241522    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-metrics-certs\") pod \"maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.241642 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.241554    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.241642 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.241581    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d05e78-8832-4213-b976-f7b5b42c0863-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.241765 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:44:33.241682    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-serving-cert: secret "maas-api-e2e-praxis-b-d22078-serving-cert" not found
Oct 31 03:44:33.241800 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:44:33.241767    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-maas-api-tls podName:86d05e78-8832-4213-b976-f7b5b42c0863 nodeName:}" failed. No retries permitted until 2026-10-31 03:44:33.741733196 +0000 UTC m=+2215.302212923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-maas-api-tls") pod "maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4" (UID: "86d05e78-8832-4213-b976-f7b5b42c0863") : secret "maas-api-e2e-praxis-b-d22078-serving-cert" not found
Oct 31 03:44:33.242368 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.242345    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d05e78-8832-4213-b976-f7b5b42c0863-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.244204 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.244186    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-metrics-certs\") pod \"maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.295861 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.295827    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97qn5\" (UniqueName: \"kubernetes.io/projected/86d05e78-8832-4213-b976-f7b5b42c0863-kube-api-access-97qn5\") pod \"maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.748259 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.748219    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.750848 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.750818    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:33.966246 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:33.966208    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:34.384538 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:34.384500    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"]
Oct 31 03:44:34.385500 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:44:34.385476    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d05e78_8832_4213_b976_f7b5b42c0863.slice/crio-689b4e87c89347cb022235617cbfb268358b419a2545c89ad4250534e19351a5 WatchSource:0}: Error finding container 689b4e87c89347cb022235617cbfb268358b419a2545c89ad4250534e19351a5: Status 404 returned error can't find the container with id 689b4e87c89347cb022235617cbfb268358b419a2545c89ad4250534e19351a5
Oct 31 03:44:34.774176 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:34.774082    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4" event={"ID":"86d05e78-8832-4213-b976-f7b5b42c0863","Type":"ContainerStarted","Data":"689b4e87c89347cb022235617cbfb268358b419a2545c89ad4250534e19351a5"}
Oct 31 03:44:35.781866 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:35.781826    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4" event={"ID":"86d05e78-8832-4213-b976-f7b5b42c0863","Type":"ContainerStarted","Data":"e1b80ec675a0df272a6ef2e25088bea58e78bb75f07fabb6ce7851e7a4eba6e8"}
Oct 31 03:44:35.782335 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:35.781938    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:35.891636 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:35.891580    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4" podStartSLOduration=1.960363616 podStartE2EDuration="2.891564457s" totalImagesPullingTime="931.200841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:35.890498878 +0000 UTC m=+2217.450978628" watchObservedRunningTime="2026-10-31 03:44:35.891564457 +0000 UTC m=+2217.452044188"
Oct 31 03:44:41.794397 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:41.794365    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:44:44.958084 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:44.958047    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"]
Oct 31 03:44:44.963396 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:44.963368    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.012937 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.012902    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-14600f-serving-cert\""
Oct 31 03:44:45.062097 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.062067    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"]
Oct 31 03:44:45.087819 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.087788    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-maas-api-tls\") pod \"maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.088003 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.087827    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-metrics-certs\") pod \"maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.088076 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.088056    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lncp\" (UniqueName: \"kubernetes.io/projected/9b94b81c-0c49-43ea-8a9d-d656802f1ade-kube-api-access-8lncp\") pod \"maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.088167 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.088149    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b94b81c-0c49-43ea-8a9d-d656802f1ade-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.188980 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.188940    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lncp\" (UniqueName: \"kubernetes.io/projected/9b94b81c-0c49-43ea-8a9d-d656802f1ade-kube-api-access-8lncp\") pod \"maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.189183 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.189012    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b94b81c-0c49-43ea-8a9d-d656802f1ade-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.189183 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.189087    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-maas-api-tls\") pod \"maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.189183 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.189109    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-metrics-certs\") pod \"maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.189303 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:44:45.189254    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-serving-cert: secret "maas-api-e2e-ipp-a-14600f-serving-cert" not found
Oct 31 03:44:45.189344 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:44:45.189333    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-maas-api-tls podName:9b94b81c-0c49-43ea-8a9d-d656802f1ade nodeName:}" failed. No retries permitted until 2026-10-31 03:44:45.689315342 +0000 UTC m=+2227.249795066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-maas-api-tls") pod "maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf" (UID: "9b94b81c-0c49-43ea-8a9d-d656802f1ade") : secret "maas-api-e2e-ipp-a-14600f-serving-cert" not found
Oct 31 03:44:45.189711 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.189684    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b94b81c-0c49-43ea-8a9d-d656802f1ade-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.191612 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.191588    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-metrics-certs\") pod \"maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.243745 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.243671    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lncp\" (UniqueName: \"kubernetes.io/projected/9b94b81c-0c49-43ea-8a9d-d656802f1ade-kube-api-access-8lncp\") pod \"maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.695267 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.695227    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-maas-api-tls\") pod \"maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.697766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.697737    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-maas-api-tls\") pod \"maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:45.877238 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:45.877202    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:46.176066 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:44:46.176030    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:44:46.248483 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:46.248455    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"]
Oct 31 03:44:46.250999 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:44:46.250967    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b94b81c_0c49_43ea_8a9d_d656802f1ade.slice/crio-5c026042324e18cce86e50c15a643bb136904c8467b1afb1ca2953e563e5f627 WatchSource:0}: Error finding container 5c026042324e18cce86e50c15a643bb136904c8467b1afb1ca2953e563e5f627: Status 404 returned error can't find the container with id 5c026042324e18cce86e50c15a643bb136904c8467b1afb1ca2953e563e5f627
Oct 31 03:44:46.852338 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:46.852291    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf" event={"ID":"9b94b81c-0c49-43ea-8a9d-d656802f1ade","Type":"ContainerStarted","Data":"5c026042324e18cce86e50c15a643bb136904c8467b1afb1ca2953e563e5f627"}
Oct 31 03:44:47.860308 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:47.860268    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf" event={"ID":"9b94b81c-0c49-43ea-8a9d-d656802f1ade","Type":"ContainerStarted","Data":"ce104525b06f97e3ae255bb18d6812aec70a457217cbda44e72552cced99165c"}
Oct 31 03:44:47.860720 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:47.860408    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:44:47.972332 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:47.972280    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf" podStartSLOduration=3.076986798 podStartE2EDuration="3.972263645s" totalImagesPullingTime="895.276847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:47.970095129 +0000 UTC m=+2229.530574859" watchObservedRunningTime="2026-10-31 03:44:47.972263645 +0000 UTC m=+2229.532743376"
Oct 31 03:44:53.872588 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:44:53.872558    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:45:00.193342 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.193291    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"]
Oct 31 03:45:00.197321 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.197302    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"
Oct 31 03:45:00.214046 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.214017    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"]
Oct 31 03:45:00.217953 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.217928    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"
Oct 31 03:45:00.265442 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.265391    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587c546c-eb63-434b-aeea-45527d431c43-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890305-lrs5n\" (UID: \"587c546c-eb63-434b-aeea-45527d431c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"
Oct 31 03:45:00.265778 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.265754    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j929w\" (UniqueName: \"kubernetes.io/projected/7fffc7b3-173d-4747-8362-aaeb3e57972b-kube-api-access-j929w\") pod \"maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl\" (UID: \"7fffc7b3-173d-4747-8362-aaeb3e57972b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"
Oct 31 03:45:00.266086 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.266056    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fffc7b3-173d-4747-8362-aaeb3e57972b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl\" (UID: \"7fffc7b3-173d-4747-8362-aaeb3e57972b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"
Oct 31 03:45:00.266265 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.266149    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg4kl\" (UniqueName: \"kubernetes.io/projected/587c546c-eb63-434b-aeea-45527d431c43-kube-api-access-lg4kl\") pod \"maas-api-key-cleanup-29890305-lrs5n\" (UID: \"587c546c-eb63-434b-aeea-45527d431c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"
Oct 31 03:45:00.271459 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.271428    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"]
Oct 31 03:45:00.279908 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.279864    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"]
Oct 31 03:45:00.321577 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.321549    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:45:00.366950 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.366847    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg4kl\" (UniqueName: \"kubernetes.io/projected/587c546c-eb63-434b-aeea-45527d431c43-kube-api-access-lg4kl\") pod \"maas-api-key-cleanup-29890305-lrs5n\" (UID: \"587c546c-eb63-434b-aeea-45527d431c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"
Oct 31 03:45:00.367129 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.367005    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587c546c-eb63-434b-aeea-45527d431c43-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890305-lrs5n\" (UID: \"587c546c-eb63-434b-aeea-45527d431c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"
Oct 31 03:45:00.367129 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.367088    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j929w\" (UniqueName: \"kubernetes.io/projected/7fffc7b3-173d-4747-8362-aaeb3e57972b-kube-api-access-j929w\") pod \"maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl\" (UID: \"7fffc7b3-173d-4747-8362-aaeb3e57972b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"
Oct 31 03:45:00.367247 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.367178    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fffc7b3-173d-4747-8362-aaeb3e57972b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl\" (UID: \"7fffc7b3-173d-4747-8362-aaeb3e57972b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"
Oct 31 03:45:00.367870 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.367843    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587c546c-eb63-434b-aeea-45527d431c43-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890305-lrs5n\" (UID: \"587c546c-eb63-434b-aeea-45527d431c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"
Oct 31 03:45:00.367983 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.367905    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fffc7b3-173d-4747-8362-aaeb3e57972b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl\" (UID: \"7fffc7b3-173d-4747-8362-aaeb3e57972b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"
Oct 31 03:45:00.421890 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.421842    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j929w\" (UniqueName: \"kubernetes.io/projected/7fffc7b3-173d-4747-8362-aaeb3e57972b-kube-api-access-j929w\") pod \"maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl\" (UID: \"7fffc7b3-173d-4747-8362-aaeb3e57972b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"
Oct 31 03:45:00.422135 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.421842    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg4kl\" (UniqueName: \"kubernetes.io/projected/587c546c-eb63-434b-aeea-45527d431c43-kube-api-access-lg4kl\") pod \"maas-api-key-cleanup-29890305-lrs5n\" (UID: \"587c546c-eb63-434b-aeea-45527d431c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"
Oct 31 03:45:00.511968 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.511916    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"
Oct 31 03:45:00.532409 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.532380    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"
Oct 31 03:45:00.888380 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.888353    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"]
Oct 31 03:45:00.890647 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:45:00.890617    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587c546c_eb63_434b_aeea_45527d431c43.slice/crio-73a40bba2addcf6e43fcbfce785d9071367508173563fc58b09bd16bde6a3d60 WatchSource:0}: Error finding container 73a40bba2addcf6e43fcbfce785d9071367508173563fc58b09bd16bde6a3d60: Status 404 returned error can't find the container with id 73a40bba2addcf6e43fcbfce785d9071367508173563fc58b09bd16bde6a3d60
Oct 31 03:45:00.932105 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.932068    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n" event={"ID":"587c546c-eb63-434b-aeea-45527d431c43","Type":"ContainerStarted","Data":"73a40bba2addcf6e43fcbfce785d9071367508173563fc58b09bd16bde6a3d60"}
Oct 31 03:45:00.935959 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:00.935936    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"]
Oct 31 03:45:00.938327 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:45:00.938296    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fffc7b3_173d_4747_8362_aaeb3e57972b.slice/crio-920bad18de802e6978cbb3a2f2b1bd8867d780fd56c5ba104b5911705d5654a0 WatchSource:0}: Error finding container 920bad18de802e6978cbb3a2f2b1bd8867d780fd56c5ba104b5911705d5654a0: Status 404 returned error can't find the container with id 920bad18de802e6978cbb3a2f2b1bd8867d780fd56c5ba104b5911705d5654a0
Oct 31 03:45:01.940549 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:01.940456    5415 generic.go:356] "Generic (PLEG): container finished" podID="7fffc7b3-173d-4747-8362-aaeb3e57972b" containerID="cd7a7fd4c0a09d8895b91c763d080d9ea0f658867f90955825968a18143bb691" exitCode=60
Oct 31 03:45:01.941035 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:01.940540    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl" event={"ID":"7fffc7b3-173d-4747-8362-aaeb3e57972b","Type":"ContainerDied","Data":"cd7a7fd4c0a09d8895b91c763d080d9ea0f658867f90955825968a18143bb691"}
Oct 31 03:45:01.941035 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:01.940605    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl" event={"ID":"7fffc7b3-173d-4747-8362-aaeb3e57972b","Type":"ContainerStarted","Data":"920bad18de802e6978cbb3a2f2b1bd8867d780fd56c5ba104b5911705d5654a0"}
Oct 31 03:45:01.941035 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:01.940816    5415 scope.go:122] "RemoveContainer" containerID="cd7a7fd4c0a09d8895b91c763d080d9ea0f658867f90955825968a18143bb691"
Oct 31 03:45:01.942727 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:01.942703    5415 generic.go:356] "Generic (PLEG): container finished" podID="587c546c-eb63-434b-aeea-45527d431c43" containerID="108dd7f2d935d2747c099cac9c5c950c9e69431c689560176bcab3747ace91c0" exitCode=60
Oct 31 03:45:01.942826 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:01.942786    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n" event={"ID":"587c546c-eb63-434b-aeea-45527d431c43","Type":"ContainerDied","Data":"108dd7f2d935d2747c099cac9c5c950c9e69431c689560176bcab3747ace91c0"}
Oct 31 03:45:01.943090 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:01.943076    5415 scope.go:122] "RemoveContainer" containerID="108dd7f2d935d2747c099cac9c5c950c9e69431c689560176bcab3747ace91c0"
Oct 31 03:45:02.951254 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:02.951211    5415 generic.go:356] "Generic (PLEG): container finished" podID="587c546c-eb63-434b-aeea-45527d431c43" containerID="9dc5b4091ad73608e5c3b8cd18029fd2c102dda0cb6d5a7906b6daa952610ec0" exitCode=60
Oct 31 03:45:02.951701 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:02.951311    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n" event={"ID":"587c546c-eb63-434b-aeea-45527d431c43","Type":"ContainerDied","Data":"9dc5b4091ad73608e5c3b8cd18029fd2c102dda0cb6d5a7906b6daa952610ec0"}
Oct 31 03:45:02.951701 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:02.951387    5415 scope.go:122] "RemoveContainer" containerID="108dd7f2d935d2747c099cac9c5c950c9e69431c689560176bcab3747ace91c0"
Oct 31 03:45:02.951809 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:02.951705    5415 scope.go:122] "RemoveContainer" containerID="9dc5b4091ad73608e5c3b8cd18029fd2c102dda0cb6d5a7906b6daa952610ec0"
Oct 31 03:45:02.952057 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:45:02.952027    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890305-lrs5n_odh-ai-gateway-infra(587c546c-eb63-434b-aeea-45527d431c43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n" podUID="587c546c-eb63-434b-aeea-45527d431c43"
Oct 31 03:45:02.953778 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:02.953756    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl" event={"ID":"7fffc7b3-173d-4747-8362-aaeb3e57972b","Type":"ContainerStarted","Data":"2b0afd42e9208b79189145392f2e7b9e27e3add708d5b0bfdeafffba03f4f60f"}
Oct 31 03:45:03.167872 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:03.167814    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl" podStartSLOduration=3.167798034 podStartE2EDuration="3.167798034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:03.166815604 +0000 UTC m=+2244.727295335" watchObservedRunningTime="2026-10-31 03:45:03.167798034 +0000 UTC m=+2244.728277765"
Oct 31 03:45:03.961343 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:03.961310    5415 generic.go:356] "Generic (PLEG): container finished" podID="7fffc7b3-173d-4747-8362-aaeb3e57972b" containerID="2b0afd42e9208b79189145392f2e7b9e27e3add708d5b0bfdeafffba03f4f60f" exitCode=60
Oct 31 03:45:03.961852 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:03.961395    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl" event={"ID":"7fffc7b3-173d-4747-8362-aaeb3e57972b","Type":"ContainerDied","Data":"2b0afd42e9208b79189145392f2e7b9e27e3add708d5b0bfdeafffba03f4f60f"}
Oct 31 03:45:03.961852 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:03.961516    5415 scope.go:122] "RemoveContainer" containerID="cd7a7fd4c0a09d8895b91c763d080d9ea0f658867f90955825968a18143bb691"
Oct 31 03:45:03.961852 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:03.961687    5415 scope.go:122] "RemoveContainer" containerID="2b0afd42e9208b79189145392f2e7b9e27e3add708d5b0bfdeafffba03f4f60f"
Oct 31 03:45:03.962065 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:45:03.962041    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl_odh-ai-gateway-infra(7fffc7b3-173d-4747-8362-aaeb3e57972b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl" podUID="7fffc7b3-173d-4747-8362-aaeb3e57972b"
Oct 31 03:45:03.963459 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:03.963440    5415 scope.go:122] "RemoveContainer" containerID="9dc5b4091ad73608e5c3b8cd18029fd2c102dda0cb6d5a7906b6daa952610ec0"
Oct 31 03:45:03.963709 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:45:03.963691    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890305-lrs5n_odh-ai-gateway-infra(587c546c-eb63-434b-aeea-45527d431c43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n" podUID="587c546c-eb63-434b-aeea-45527d431c43"
Oct 31 03:45:04.971145 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:04.971110    5415 scope.go:122] "RemoveContainer" containerID="2b0afd42e9208b79189145392f2e7b9e27e3add708d5b0bfdeafffba03f4f60f"
Oct 31 03:45:04.971627 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:45:04.971463    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl_odh-ai-gateway-infra(7fffc7b3-173d-4747-8362-aaeb3e57972b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl" podUID="7fffc7b3-173d-4747-8362-aaeb3e57972b"
Oct 31 03:45:15.599185 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:15.599107    5415 scope.go:122] "RemoveContainer" containerID="2b0afd42e9208b79189145392f2e7b9e27e3add708d5b0bfdeafffba03f4f60f"
Oct 31 03:45:16.042996 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:16.042956    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl" event={"ID":"7fffc7b3-173d-4747-8362-aaeb3e57972b","Type":"ContainerStarted","Data":"be66070a6d2be7db6e809776e83147376567abfa98d93d8e540389dab22afc00"}
Oct 31 03:45:16.176648 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:45:16.176562    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:45:16.599485 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:16.599457    5415 scope.go:122] "RemoveContainer" containerID="9dc5b4091ad73608e5c3b8cd18029fd2c102dda0cb6d5a7906b6daa952610ec0"
Oct 31 03:45:17.050858 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:17.050820    5415 generic.go:356] "Generic (PLEG): container finished" podID="7fffc7b3-173d-4747-8362-aaeb3e57972b" containerID="be66070a6d2be7db6e809776e83147376567abfa98d93d8e540389dab22afc00" exitCode=60
Oct 31 03:45:17.051067 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:17.050914    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl" event={"ID":"7fffc7b3-173d-4747-8362-aaeb3e57972b","Type":"ContainerDied","Data":"be66070a6d2be7db6e809776e83147376567abfa98d93d8e540389dab22afc00"}
Oct 31 03:45:17.051067 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:17.050983    5415 scope.go:122] "RemoveContainer" containerID="2b0afd42e9208b79189145392f2e7b9e27e3add708d5b0bfdeafffba03f4f60f"
Oct 31 03:45:17.051401 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:17.051378    5415 scope.go:122] "RemoveContainer" containerID="be66070a6d2be7db6e809776e83147376567abfa98d93d8e540389dab22afc00"
Oct 31 03:45:17.051740 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:45:17.051718    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl_odh-ai-gateway-infra(7fffc7b3-173d-4747-8362-aaeb3e57972b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl" podUID="7fffc7b3-173d-4747-8362-aaeb3e57972b"
Oct 31 03:45:17.053122 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:17.053092    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n" event={"ID":"587c546c-eb63-434b-aeea-45527d431c43","Type":"ContainerStarted","Data":"d22a8c8f5c94170d200e84255995b3886fe1f3a2c762586933d3d9cadb227104"}
Oct 31 03:45:17.160430 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:17.160344    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"]
Oct 31 03:45:17.287901 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:17.287829    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n" podStartSLOduration=17.28781404 podStartE2EDuration="17.28781404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:17.284989688 +0000 UTC m=+2258.845469419" watchObservedRunningTime="2026-10-31 03:45:17.28781404 +0000 UTC m=+2258.848293770"
Oct 31 03:45:18.063314 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:18.063271    5415 generic.go:356] "Generic (PLEG): container finished" podID="587c546c-eb63-434b-aeea-45527d431c43" containerID="d22a8c8f5c94170d200e84255995b3886fe1f3a2c762586933d3d9cadb227104" exitCode=60
Oct 31 03:45:18.063749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:18.063325    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n" event={"ID":"587c546c-eb63-434b-aeea-45527d431c43","Type":"ContainerDied","Data":"d22a8c8f5c94170d200e84255995b3886fe1f3a2c762586933d3d9cadb227104"}
Oct 31 03:45:18.063749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:18.063386    5415 scope.go:122] "RemoveContainer" containerID="9dc5b4091ad73608e5c3b8cd18029fd2c102dda0cb6d5a7906b6daa952610ec0"
Oct 31 03:45:18.063868 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:18.063847    5415 scope.go:122] "RemoveContainer" containerID="d22a8c8f5c94170d200e84255995b3886fe1f3a2c762586933d3d9cadb227104"
Oct 31 03:45:18.064254 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:45:18.064225    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890305-lrs5n_odh-ai-gateway-infra(587c546c-eb63-434b-aeea-45527d431c43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n" podUID="587c546c-eb63-434b-aeea-45527d431c43"
Oct 31 03:45:18.203209 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:18.203179    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"
Oct 31 03:45:18.316421 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:18.316333    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"]
Oct 31 03:45:18.381543 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:18.381509    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fffc7b3-173d-4747-8362-aaeb3e57972b-kube-api-access-j929w\" (UniqueName: \"kubernetes.io/projected/7fffc7b3-173d-4747-8362-aaeb3e57972b-kube-api-access-j929w\") pod \"7fffc7b3-173d-4747-8362-aaeb3e57972b\" (UID: \"7fffc7b3-173d-4747-8362-aaeb3e57972b\") "
Oct 31 03:45:18.381754 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:18.381551    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fffc7b3-173d-4747-8362-aaeb3e57972b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fffc7b3-173d-4747-8362-aaeb3e57972b-maas-api-service-ca\") pod \"7fffc7b3-173d-4747-8362-aaeb3e57972b\" (UID: \"7fffc7b3-173d-4747-8362-aaeb3e57972b\") "
Oct 31 03:45:18.381950 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:18.381927    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fffc7b3-173d-4747-8362-aaeb3e57972b-maas-api-service-ca" pod "7fffc7b3-173d-4747-8362-aaeb3e57972b" (UID: "7fffc7b3-173d-4747-8362-aaeb3e57972b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:18.383711 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:18.383692    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fffc7b3-173d-4747-8362-aaeb3e57972b-kube-api-access-j929w" pod "7fffc7b3-173d-4747-8362-aaeb3e57972b" (UID: "7fffc7b3-173d-4747-8362-aaeb3e57972b"). InnerVolumeSpecName "kube-api-access-j929w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:18.482600 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:18.482562    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j929w\" (UniqueName: \"kubernetes.io/projected/7fffc7b3-173d-4747-8362-aaeb3e57972b-kube-api-access-j929w\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:18.482600 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:18.482593    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fffc7b3-173d-4747-8362-aaeb3e57972b-maas-api-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:19.070808 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:19.070729    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"
Oct 31 03:45:19.070808 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:19.070745    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl" event={"ID":"7fffc7b3-173d-4747-8362-aaeb3e57972b","Type":"ContainerDied","Data":"920bad18de802e6978cbb3a2f2b1bd8867d780fd56c5ba104b5911705d5654a0"}
Oct 31 03:45:19.070808 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:19.070810    5415 scope.go:122] "RemoveContainer" containerID="be66070a6d2be7db6e809776e83147376567abfa98d93d8e540389dab22afc00"
Oct 31 03:45:19.224940 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:19.224913    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"
Oct 31 03:45:19.349573 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:19.349539    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"]
Oct 31 03:45:19.353073 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:19.353043    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d22078-29890305-gn9rl"]
Oct 31 03:45:19.393265 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:19.393230    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587c546c-eb63-434b-aeea-45527d431c43-kube-api-access-lg4kl\" (UniqueName: \"kubernetes.io/projected/587c546c-eb63-434b-aeea-45527d431c43-kube-api-access-lg4kl\") pod \"587c546c-eb63-434b-aeea-45527d431c43\" (UID: \"587c546c-eb63-434b-aeea-45527d431c43\") "
Oct 31 03:45:19.393480 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:19.393377    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587c546c-eb63-434b-aeea-45527d431c43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587c546c-eb63-434b-aeea-45527d431c43-maas-api-service-ca\") pod \"587c546c-eb63-434b-aeea-45527d431c43\" (UID: \"587c546c-eb63-434b-aeea-45527d431c43\") "
Oct 31 03:45:19.393778 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:19.393730    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587c546c-eb63-434b-aeea-45527d431c43-maas-api-service-ca" pod "587c546c-eb63-434b-aeea-45527d431c43" (UID: "587c546c-eb63-434b-aeea-45527d431c43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:19.393915 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:19.393842    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587c546c-eb63-434b-aeea-45527d431c43-maas-api-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:19.395523 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:19.395504    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587c546c-eb63-434b-aeea-45527d431c43-kube-api-access-lg4kl" pod "587c546c-eb63-434b-aeea-45527d431c43" (UID: "587c546c-eb63-434b-aeea-45527d431c43"). InnerVolumeSpecName "kube-api-access-lg4kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:19.495073 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:19.495039    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg4kl\" (UniqueName: \"kubernetes.io/projected/587c546c-eb63-434b-aeea-45527d431c43-kube-api-access-lg4kl\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:20.081753 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:20.081722    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"
Oct 31 03:45:20.081753 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:20.081751    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n" event={"ID":"587c546c-eb63-434b-aeea-45527d431c43","Type":"ContainerDied","Data":"73a40bba2addcf6e43fcbfce785d9071367508173563fc58b09bd16bde6a3d60"}
Oct 31 03:45:20.082330 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:20.081820    5415 scope.go:122] "RemoveContainer" containerID="d22a8c8f5c94170d200e84255995b3886fe1f3a2c762586933d3d9cadb227104"
Oct 31 03:45:20.238106 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:20.238074    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"]
Oct 31 03:45:20.240209 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:20.240181    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-lrs5n"]
Oct 31 03:45:20.605929 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:20.605892    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587c546c-eb63-434b-aeea-45527d431c43" path="/var/lib/kubelet/pods/587c546c-eb63-434b-aeea-45527d431c43/volumes"
Oct 31 03:45:20.606347 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:45:20.606333    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fffc7b3-173d-4747-8362-aaeb3e57972b" path="/var/lib/kubelet/pods/7fffc7b3-173d-4747-8362-aaeb3e57972b/volumes"
Oct 31 03:45:46.176054 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:45:46.176014    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:46:01.049276 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:01.049237    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"]
Oct 31 03:46:16.070599 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.070551    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp" podUID="87788dd8-2aa4-4b65-80f2-8116be88a903" containerName="main" containerID="cri-o://a14fa56e76e1d20e441fef56c8e77acc98089feb60b039d6ef5d093f6d44ca5e" gracePeriod=45
Oct 31 03:46:16.182249 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:46:16.182212    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:46:16.356349 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.356321    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:46:16.423262 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.423227    5415 generic.go:356] "Generic (PLEG): container finished" podID="87788dd8-2aa4-4b65-80f2-8116be88a903" containerID="a14fa56e76e1d20e441fef56c8e77acc98089feb60b039d6ef5d093f6d44ca5e" exitCode=0
Oct 31 03:46:16.423416 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.423277    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"
Oct 31 03:46:16.423416 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.423318    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp" event={"ID":"87788dd8-2aa4-4b65-80f2-8116be88a903","Type":"ContainerDied","Data":"a14fa56e76e1d20e441fef56c8e77acc98089feb60b039d6ef5d093f6d44ca5e"}
Oct 31 03:46:16.423416 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.423368    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp" event={"ID":"87788dd8-2aa4-4b65-80f2-8116be88a903","Type":"ContainerDied","Data":"bef7f2dcac8752e69029ba88ec54a9fd9fdd4fc1f7d4e32219ba6c23abbf0975"}
Oct 31 03:46:16.423416 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.423386    5415 scope.go:122] "RemoveContainer" containerID="a14fa56e76e1d20e441fef56c8e77acc98089feb60b039d6ef5d093f6d44ca5e"
Oct 31 03:46:16.432731 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.432710    5415 scope.go:122] "RemoveContainer" containerID="a14fa56e76e1d20e441fef56c8e77acc98089feb60b039d6ef5d093f6d44ca5e"
Oct 31 03:46:16.433136 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:46:16.433109    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14fa56e76e1d20e441fef56c8e77acc98089feb60b039d6ef5d093f6d44ca5e\": container with ID starting with a14fa56e76e1d20e441fef56c8e77acc98089feb60b039d6ef5d093f6d44ca5e not found: ID does not exist" containerID="a14fa56e76e1d20e441fef56c8e77acc98089feb60b039d6ef5d093f6d44ca5e"
Oct 31 03:46:16.433188 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.433149    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14fa56e76e1d20e441fef56c8e77acc98089feb60b039d6ef5d093f6d44ca5e"} err="failed to get container status \"a14fa56e76e1d20e441fef56c8e77acc98089feb60b039d6ef5d093f6d44ca5e\": rpc error: code = NotFound desc = could not find container \"a14fa56e76e1d20e441fef56c8e77acc98089feb60b039d6ef5d093f6d44ca5e\": container with ID starting with a14fa56e76e1d20e441fef56c8e77acc98089feb60b039d6ef5d093f6d44ca5e not found: ID does not exist"
Oct 31 03:46:16.456972 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.456936    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87788dd8-2aa4-4b65-80f2-8116be88a903-tls-certs\" (UniqueName: \"kubernetes.io/secret/87788dd8-2aa4-4b65-80f2-8116be88a903-tls-certs\") pod \"87788dd8-2aa4-4b65-80f2-8116be88a903\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") "
Oct 31 03:46:16.457148 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.457024    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-home\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-home\") pod \"87788dd8-2aa4-4b65-80f2-8116be88a903\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") "
Oct 31 03:46:16.457148 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.457094    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-model-cache\") pod \"87788dd8-2aa4-4b65-80f2-8116be88a903\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") "
Oct 31 03:46:16.457148 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.457134    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-tmp-dir\") pod \"87788dd8-2aa4-4b65-80f2-8116be88a903\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") "
Oct 31 03:46:16.457357 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.457175    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87788dd8-2aa4-4b65-80f2-8116be88a903-kube-api-access-d97wh\" (UniqueName: \"kubernetes.io/projected/87788dd8-2aa4-4b65-80f2-8116be88a903-kube-api-access-d97wh\") pod \"87788dd8-2aa4-4b65-80f2-8116be88a903\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") "
Oct 31 03:46:16.457357 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.457217    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-dshm\") pod \"87788dd8-2aa4-4b65-80f2-8116be88a903\" (UID: \"87788dd8-2aa4-4b65-80f2-8116be88a903\") "
Oct 31 03:46:16.457357 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.457312    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-home" pod "87788dd8-2aa4-4b65-80f2-8116be88a903" (UID: "87788dd8-2aa4-4b65-80f2-8116be88a903"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:16.457525 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.457390    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-tmp-dir" pod "87788dd8-2aa4-4b65-80f2-8116be88a903" (UID: "87788dd8-2aa4-4b65-80f2-8116be88a903"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:16.457525 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.457467    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-model-cache" pod "87788dd8-2aa4-4b65-80f2-8116be88a903" (UID: "87788dd8-2aa4-4b65-80f2-8116be88a903"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:16.457677 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.457659    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:16.457716 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.457682    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:16.457716 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.457696    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:16.459364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.459332    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-dshm" pod "87788dd8-2aa4-4b65-80f2-8116be88a903" (UID: "87788dd8-2aa4-4b65-80f2-8116be88a903"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:16.459506 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.459434    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87788dd8-2aa4-4b65-80f2-8116be88a903-tls-certs" pod "87788dd8-2aa4-4b65-80f2-8116be88a903" (UID: "87788dd8-2aa4-4b65-80f2-8116be88a903"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:46:16.459506 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.459434    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87788dd8-2aa4-4b65-80f2-8116be88a903-kube-api-access-d97wh" pod "87788dd8-2aa4-4b65-80f2-8116be88a903" (UID: "87788dd8-2aa4-4b65-80f2-8116be88a903"). InnerVolumeSpecName "kube-api-access-d97wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:16.558804 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.558773    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d97wh\" (UniqueName: \"kubernetes.io/projected/87788dd8-2aa4-4b65-80f2-8116be88a903-kube-api-access-d97wh\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:16.558804 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.558802    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87788dd8-2aa4-4b65-80f2-8116be88a903-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:16.558804 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.558813    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87788dd8-2aa4-4b65-80f2-8116be88a903-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:16.831189 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.831154    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"]
Oct 31 03:46:16.837589 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:16.837556    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a32cdb/ipp-route-a32cdb-kserve-cfbc9f797-snmsp"]
Oct 31 03:46:18.606240 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:18.606198    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87788dd8-2aa4-4b65-80f2-8116be88a903" path="/var/lib/kubelet/pods/87788dd8-2aa4-4b65-80f2-8116be88a903/volumes"
Oct 31 03:46:39.914667 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:39.914629    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"]
Oct 31 03:46:39.915079 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:39.914844    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh" podUID="156d2f57-24b3-43a4-9a06-3bfa3a7f4df6" containerName="maas-api" containerID="cri-o://d216e5e84c754fe2e60fc696888636f578a0881b31fe42e1dbfe6d95e73f430b" gracePeriod=30
Oct 31 03:46:40.215624 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.215592    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:46:40.249117 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.249076    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-kube-api-access-mnb7w\" (UniqueName: \"kubernetes.io/projected/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-kube-api-access-mnb7w\") pod \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") "
Oct 31 03:46:40.249473 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.249452    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-metrics-certs\") pod \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") "
Oct 31 03:46:40.249650 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.249611    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-maas-api-tls\") pod \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") "
Oct 31 03:46:40.249823 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.249807    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-openshift-service-ca\") pod \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\" (UID: \"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6\") "
Oct 31 03:46:40.250757 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.250734    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-openshift-service-ca" pod "156d2f57-24b3-43a4-9a06-3bfa3a7f4df6" (UID: "156d2f57-24b3-43a4-9a06-3bfa3a7f4df6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:40.254183 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.254131    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-kube-api-access-mnb7w" pod "156d2f57-24b3-43a4-9a06-3bfa3a7f4df6" (UID: "156d2f57-24b3-43a4-9a06-3bfa3a7f4df6"). InnerVolumeSpecName "kube-api-access-mnb7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:40.262542 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.261046    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-maas-api-tls" pod "156d2f57-24b3-43a4-9a06-3bfa3a7f4df6" (UID: "156d2f57-24b3-43a4-9a06-3bfa3a7f4df6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:46:40.281897 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.281087    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-metrics-certs" pod "156d2f57-24b3-43a4-9a06-3bfa3a7f4df6" (UID: "156d2f57-24b3-43a4-9a06-3bfa3a7f4df6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:46:40.352577 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.352543    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:40.352796 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.352776    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:40.352940 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.352922    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:40.353072 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.353046    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnb7w\" (UniqueName: \"kubernetes.io/projected/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6-kube-api-access-mnb7w\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:40.566326 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.566244    5415 generic.go:356] "Generic (PLEG): container finished" podID="156d2f57-24b3-43a4-9a06-3bfa3a7f4df6" containerID="d216e5e84c754fe2e60fc696888636f578a0881b31fe42e1dbfe6d95e73f430b" exitCode=0
Oct 31 03:46:40.566326 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.566283    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh" event={"ID":"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6","Type":"ContainerDied","Data":"d216e5e84c754fe2e60fc696888636f578a0881b31fe42e1dbfe6d95e73f430b"}
Oct 31 03:46:40.566326 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.566290    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"
Oct 31 03:46:40.566561 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.566360    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh" event={"ID":"156d2f57-24b3-43a4-9a06-3bfa3a7f4df6","Type":"ContainerDied","Data":"07a1ee2a497d744795e4f4bf04f70f96d297c865bf8be8dab5f1abb09f79a809"}
Oct 31 03:46:40.566561 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.566385    5415 scope.go:122] "RemoveContainer" containerID="d216e5e84c754fe2e60fc696888636f578a0881b31fe42e1dbfe6d95e73f430b"
Oct 31 03:46:40.576167 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.576145    5415 scope.go:122] "RemoveContainer" containerID="d216e5e84c754fe2e60fc696888636f578a0881b31fe42e1dbfe6d95e73f430b"
Oct 31 03:46:40.576458 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:46:40.576439    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d216e5e84c754fe2e60fc696888636f578a0881b31fe42e1dbfe6d95e73f430b\": container with ID starting with d216e5e84c754fe2e60fc696888636f578a0881b31fe42e1dbfe6d95e73f430b not found: ID does not exist" containerID="d216e5e84c754fe2e60fc696888636f578a0881b31fe42e1dbfe6d95e73f430b"
Oct 31 03:46:40.576507 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.576469    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d216e5e84c754fe2e60fc696888636f578a0881b31fe42e1dbfe6d95e73f430b"} err="failed to get container status \"d216e5e84c754fe2e60fc696888636f578a0881b31fe42e1dbfe6d95e73f430b\": rpc error: code = NotFound desc = could not find container \"d216e5e84c754fe2e60fc696888636f578a0881b31fe42e1dbfe6d95e73f430b\": container with ID starting with d216e5e84c754fe2e60fc696888636f578a0881b31fe42e1dbfe6d95e73f430b not found: ID does not exist"
Oct 31 03:46:40.720478 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.720444    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"]
Oct 31 03:46:40.724280 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:40.724248    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6f9fd2-6b77f76c49-x9wfh"]
Oct 31 03:46:41.479854 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:41.479817    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"]
Oct 31 03:46:41.578669 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:41.578635    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"]
Oct 31 03:46:42.606594 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:42.606564    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156d2f57-24b3-43a4-9a06-3bfa3a7f4df6" path="/var/lib/kubelet/pods/156d2f57-24b3-43a4-9a06-3bfa3a7f4df6/volumes"
Oct 31 03:46:46.175553 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:46:46.175425    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:46:46.480735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.480617    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x" podUID="8c780d3a-6662-4c40-b816-3cf1f6da41ff" containerName="payload-processing" containerID="cri-o://dd07e4a1b44467e8c3ae58f85c84e6262da6a18dd167c55f2e4890e8c97f313b" gracePeriod=25
Oct 31 03:46:46.579639 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.579575    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch" podUID="da356000-9c44-4fa1-9d31-0dd6e65145f4" containerName="payload-pre-processing" containerID="cri-o://ed655209e105418907f381a8204d2cc81fc8fd27d3c87736c26a192fb022dfca" gracePeriod=25
Oct 31 03:46:46.602289 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.602259    5415 generic.go:356] "Generic (PLEG): container finished" podID="8c780d3a-6662-4c40-b816-3cf1f6da41ff" containerID="dd07e4a1b44467e8c3ae58f85c84e6262da6a18dd167c55f2e4890e8c97f313b" exitCode=0
Oct 31 03:46:46.607303 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.607273    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x" event={"ID":"8c780d3a-6662-4c40-b816-3cf1f6da41ff","Type":"ContainerDied","Data":"dd07e4a1b44467e8c3ae58f85c84e6262da6a18dd167c55f2e4890e8c97f313b"}
Oct 31 03:46:46.755840 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.755806    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"
Oct 31 03:46:46.830793 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.830751    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c780d3a-6662-4c40-b816-3cf1f6da41ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c780d3a-6662-4c40-b816-3cf1f6da41ff-plugins-config-volume\") pod \"8c780d3a-6662-4c40-b816-3cf1f6da41ff\" (UID: \"8c780d3a-6662-4c40-b816-3cf1f6da41ff\") "
Oct 31 03:46:46.831006 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.830836    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c780d3a-6662-4c40-b816-3cf1f6da41ff-kube-api-access-rk7qc\" (UniqueName: \"kubernetes.io/projected/8c780d3a-6662-4c40-b816-3cf1f6da41ff-kube-api-access-rk7qc\") pod \"8c780d3a-6662-4c40-b816-3cf1f6da41ff\" (UID: \"8c780d3a-6662-4c40-b816-3cf1f6da41ff\") "
Oct 31 03:46:46.831203 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.831179    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c780d3a-6662-4c40-b816-3cf1f6da41ff-plugins-config-volume" pod "8c780d3a-6662-4c40-b816-3cf1f6da41ff" (UID: "8c780d3a-6662-4c40-b816-3cf1f6da41ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:46.831352 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.831338    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c780d3a-6662-4c40-b816-3cf1f6da41ff-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:46.833013 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.832983    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c780d3a-6662-4c40-b816-3cf1f6da41ff-kube-api-access-rk7qc" pod "8c780d3a-6662-4c40-b816-3cf1f6da41ff" (UID: "8c780d3a-6662-4c40-b816-3cf1f6da41ff"). InnerVolumeSpecName "kube-api-access-rk7qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:46.860714 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.860688    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"
Oct 31 03:46:46.931955 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.931917    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da356000-9c44-4fa1-9d31-0dd6e65145f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da356000-9c44-4fa1-9d31-0dd6e65145f4-plugins-config-volume\") pod \"da356000-9c44-4fa1-9d31-0dd6e65145f4\" (UID: \"da356000-9c44-4fa1-9d31-0dd6e65145f4\") "
Oct 31 03:46:46.932141 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.932088    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da356000-9c44-4fa1-9d31-0dd6e65145f4-kube-api-access-52hcb\" (UniqueName: \"kubernetes.io/projected/da356000-9c44-4fa1-9d31-0dd6e65145f4-kube-api-access-52hcb\") pod \"da356000-9c44-4fa1-9d31-0dd6e65145f4\" (UID: \"da356000-9c44-4fa1-9d31-0dd6e65145f4\") "
Oct 31 03:46:46.932326 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.932300    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da356000-9c44-4fa1-9d31-0dd6e65145f4-plugins-config-volume" pod "da356000-9c44-4fa1-9d31-0dd6e65145f4" (UID: "da356000-9c44-4fa1-9d31-0dd6e65145f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:46.932505 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.932481    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk7qc\" (UniqueName: \"kubernetes.io/projected/8c780d3a-6662-4c40-b816-3cf1f6da41ff-kube-api-access-rk7qc\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:46.932505 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.932500    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da356000-9c44-4fa1-9d31-0dd6e65145f4-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:46.934531 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:46.934510    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da356000-9c44-4fa1-9d31-0dd6e65145f4-kube-api-access-52hcb" pod "da356000-9c44-4fa1-9d31-0dd6e65145f4" (UID: "da356000-9c44-4fa1-9d31-0dd6e65145f4"). InnerVolumeSpecName "kube-api-access-52hcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:47.034022 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.033920    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52hcb\" (UniqueName: \"kubernetes.io/projected/da356000-9c44-4fa1-9d31-0dd6e65145f4-kube-api-access-52hcb\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:47.609586 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.609555    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"
Oct 31 03:46:47.610165 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.609600    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x" event={"ID":"8c780d3a-6662-4c40-b816-3cf1f6da41ff","Type":"ContainerDied","Data":"591862ffe2d5b805208cbcdbfca10f456f724cc9ce16a8d857d4ff44578e3122"}
Oct 31 03:46:47.610165 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.609680    5415 scope.go:122] "RemoveContainer" containerID="dd07e4a1b44467e8c3ae58f85c84e6262da6a18dd167c55f2e4890e8c97f313b"
Oct 31 03:46:47.611429 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.611396    5415 generic.go:356] "Generic (PLEG): container finished" podID="da356000-9c44-4fa1-9d31-0dd6e65145f4" containerID="ed655209e105418907f381a8204d2cc81fc8fd27d3c87736c26a192fb022dfca" exitCode=0
Oct 31 03:46:47.611579 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.611433    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"
Oct 31 03:46:47.611579 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.611483    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch" event={"ID":"da356000-9c44-4fa1-9d31-0dd6e65145f4","Type":"ContainerDied","Data":"ed655209e105418907f381a8204d2cc81fc8fd27d3c87736c26a192fb022dfca"}
Oct 31 03:46:47.611579 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.611536    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch" event={"ID":"da356000-9c44-4fa1-9d31-0dd6e65145f4","Type":"ContainerDied","Data":"d291bfb3784d3a2d957103395daebe820184347fd70d3d4df0e0bee6571a9290"}
Oct 31 03:46:47.621114 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.621093    5415 scope.go:122] "RemoveContainer" containerID="ed655209e105418907f381a8204d2cc81fc8fd27d3c87736c26a192fb022dfca"
Oct 31 03:46:47.632452 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.632429    5415 scope.go:122] "RemoveContainer" containerID="ed655209e105418907f381a8204d2cc81fc8fd27d3c87736c26a192fb022dfca"
Oct 31 03:46:47.632794 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:46:47.632768    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed655209e105418907f381a8204d2cc81fc8fd27d3c87736c26a192fb022dfca\": container with ID starting with ed655209e105418907f381a8204d2cc81fc8fd27d3c87736c26a192fb022dfca not found: ID does not exist" containerID="ed655209e105418907f381a8204d2cc81fc8fd27d3c87736c26a192fb022dfca"
Oct 31 03:46:47.632856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.632804    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed655209e105418907f381a8204d2cc81fc8fd27d3c87736c26a192fb022dfca"} err="failed to get container status \"ed655209e105418907f381a8204d2cc81fc8fd27d3c87736c26a192fb022dfca\": rpc error: code = NotFound desc = could not find container \"ed655209e105418907f381a8204d2cc81fc8fd27d3c87736c26a192fb022dfca\": container with ID starting with ed655209e105418907f381a8204d2cc81fc8fd27d3c87736c26a192fb022dfca not found: ID does not exist"
Oct 31 03:46:47.765340 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.765255    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"]
Oct 31 03:46:47.770296 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.770268    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-03d737-697d54c9cf-ft2ch"]
Oct 31 03:46:47.873779 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.873740    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"]
Oct 31 03:46:47.877423 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:47.877381    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-03d737-7b9f454cf7-b6x5x"]
Oct 31 03:46:48.605472 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:48.605437    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c780d3a-6662-4c40-b816-3cf1f6da41ff" path="/var/lib/kubelet/pods/8c780d3a-6662-4c40-b816-3cf1f6da41ff/volumes"
Oct 31 03:46:48.605887 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:48.605861    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da356000-9c44-4fa1-9d31-0dd6e65145f4" path="/var/lib/kubelet/pods/da356000-9c44-4fa1-9d31-0dd6e65145f4/volumes"
Oct 31 03:46:52.843222 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:52.843187    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"]
Oct 31 03:46:57.126865 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.126820    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"]
Oct 31 03:46:57.127382 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.127057    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4" podUID="86d05e78-8832-4213-b976-f7b5b42c0863" containerName="maas-api" containerID="cri-o://e1b80ec675a0df272a6ef2e25088bea58e78bb75f07fabb6ce7851e7a4eba6e8" gracePeriod=30
Oct 31 03:46:57.424055 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.424024    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:46:57.564394 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.564350    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-metrics-certs\") pod \"86d05e78-8832-4213-b976-f7b5b42c0863\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") "
Oct 31 03:46:57.564608 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.564522    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d05e78-8832-4213-b976-f7b5b42c0863-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d05e78-8832-4213-b976-f7b5b42c0863-openshift-service-ca\") pod \"86d05e78-8832-4213-b976-f7b5b42c0863\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") "
Oct 31 03:46:57.564608 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.564579    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d05e78-8832-4213-b976-f7b5b42c0863-kube-api-access-97qn5\" (UniqueName: \"kubernetes.io/projected/86d05e78-8832-4213-b976-f7b5b42c0863-kube-api-access-97qn5\") pod \"86d05e78-8832-4213-b976-f7b5b42c0863\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") "
Oct 31 03:46:57.564742 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.564621    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-maas-api-tls\") pod \"86d05e78-8832-4213-b976-f7b5b42c0863\" (UID: \"86d05e78-8832-4213-b976-f7b5b42c0863\") "
Oct 31 03:46:57.565846 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.565818    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d05e78-8832-4213-b976-f7b5b42c0863-openshift-service-ca" pod "86d05e78-8832-4213-b976-f7b5b42c0863" (UID: "86d05e78-8832-4213-b976-f7b5b42c0863"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:57.579485 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.579439    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d05e78-8832-4213-b976-f7b5b42c0863-kube-api-access-97qn5" pod "86d05e78-8832-4213-b976-f7b5b42c0863" (UID: "86d05e78-8832-4213-b976-f7b5b42c0863"). InnerVolumeSpecName "kube-api-access-97qn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:57.594416 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.594379    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-maas-api-tls" pod "86d05e78-8832-4213-b976-f7b5b42c0863" (UID: "86d05e78-8832-4213-b976-f7b5b42c0863"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:46:57.596050 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.596021    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-metrics-certs" pod "86d05e78-8832-4213-b976-f7b5b42c0863" (UID: "86d05e78-8832-4213-b976-f7b5b42c0863"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:46:57.666074 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.665988    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d05e78-8832-4213-b976-f7b5b42c0863-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:57.666074 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.666024    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97qn5\" (UniqueName: \"kubernetes.io/projected/86d05e78-8832-4213-b976-f7b5b42c0863-kube-api-access-97qn5\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:57.666074 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.666040    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:57.666074 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.666056    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86d05e78-8832-4213-b976-f7b5b42c0863-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:57.670442 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.670416    5415 generic.go:356] "Generic (PLEG): container finished" podID="86d05e78-8832-4213-b976-f7b5b42c0863" containerID="e1b80ec675a0df272a6ef2e25088bea58e78bb75f07fabb6ce7851e7a4eba6e8" exitCode=0
Oct 31 03:46:57.670590 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.670463    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"
Oct 31 03:46:57.670590 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.670507    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4" event={"ID":"86d05e78-8832-4213-b976-f7b5b42c0863","Type":"ContainerDied","Data":"e1b80ec675a0df272a6ef2e25088bea58e78bb75f07fabb6ce7851e7a4eba6e8"}
Oct 31 03:46:57.670590 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.670565    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4" event={"ID":"86d05e78-8832-4213-b976-f7b5b42c0863","Type":"ContainerDied","Data":"689b4e87c89347cb022235617cbfb268358b419a2545c89ad4250534e19351a5"}
Oct 31 03:46:57.670590 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.670588    5415 scope.go:122] "RemoveContainer" containerID="e1b80ec675a0df272a6ef2e25088bea58e78bb75f07fabb6ce7851e7a4eba6e8"
Oct 31 03:46:57.682797 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.682770    5415 scope.go:122] "RemoveContainer" containerID="e1b80ec675a0df272a6ef2e25088bea58e78bb75f07fabb6ce7851e7a4eba6e8"
Oct 31 03:46:57.683141 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:46:57.683118    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b80ec675a0df272a6ef2e25088bea58e78bb75f07fabb6ce7851e7a4eba6e8\": container with ID starting with e1b80ec675a0df272a6ef2e25088bea58e78bb75f07fabb6ce7851e7a4eba6e8 not found: ID does not exist" containerID="e1b80ec675a0df272a6ef2e25088bea58e78bb75f07fabb6ce7851e7a4eba6e8"
Oct 31 03:46:57.683248 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.683147    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b80ec675a0df272a6ef2e25088bea58e78bb75f07fabb6ce7851e7a4eba6e8"} err="failed to get container status \"e1b80ec675a0df272a6ef2e25088bea58e78bb75f07fabb6ce7851e7a4eba6e8\": rpc error: code = NotFound desc = could not find container \"e1b80ec675a0df272a6ef2e25088bea58e78bb75f07fabb6ce7851e7a4eba6e8\": container with ID starting with e1b80ec675a0df272a6ef2e25088bea58e78bb75f07fabb6ce7851e7a4eba6e8 not found: ID does not exist"
Oct 31 03:46:57.826542 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.826445    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"]
Oct 31 03:46:57.829389 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.829357    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d22078-5564f4745b-ms9z4"]
Oct 31 03:46:57.843500 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:57.843458    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42" podUID="b529469f-b83f-41af-a9a3-35f55b1eab48" containerName="payload-processing" containerID="cri-o://f7404e3d1aee147c2a82dfda365c5181c2ed29bba12b702b8e69a2414e183e01" gracePeriod=25
Oct 31 03:46:58.098240 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.098213    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"
Oct 31 03:46:58.273576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.273488    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b529469f-b83f-41af-a9a3-35f55b1eab48-kube-api-access-wdxnf\" (UniqueName: \"kubernetes.io/projected/b529469f-b83f-41af-a9a3-35f55b1eab48-kube-api-access-wdxnf\") pod \"b529469f-b83f-41af-a9a3-35f55b1eab48\" (UID: \"b529469f-b83f-41af-a9a3-35f55b1eab48\") "
Oct 31 03:46:58.273576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.273563    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b529469f-b83f-41af-a9a3-35f55b1eab48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b529469f-b83f-41af-a9a3-35f55b1eab48-plugins-config-volume\") pod \"b529469f-b83f-41af-a9a3-35f55b1eab48\" (UID: \"b529469f-b83f-41af-a9a3-35f55b1eab48\") "
Oct 31 03:46:58.274059 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.273920    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b529469f-b83f-41af-a9a3-35f55b1eab48-plugins-config-volume" pod "b529469f-b83f-41af-a9a3-35f55b1eab48" (UID: "b529469f-b83f-41af-a9a3-35f55b1eab48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:58.275610 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.275589    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b529469f-b83f-41af-a9a3-35f55b1eab48-kube-api-access-wdxnf" pod "b529469f-b83f-41af-a9a3-35f55b1eab48" (UID: "b529469f-b83f-41af-a9a3-35f55b1eab48"). InnerVolumeSpecName "kube-api-access-wdxnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:58.368747 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.368708    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"]
Oct 31 03:46:58.368952 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.368927    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955" podUID="b5272b1e-7cdc-4d44-a658-50f259988c1e" containerName="maas-api" containerID="cri-o://bb7255c5d8676971d0e04577e6b9dc3760efb94cf4390fc775f0dbfc7ae23c40" gracePeriod=30
Oct 31 03:46:58.375265 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.375236    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdxnf\" (UniqueName: \"kubernetes.io/projected/b529469f-b83f-41af-a9a3-35f55b1eab48-kube-api-access-wdxnf\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:58.375387 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.375268    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b529469f-b83f-41af-a9a3-35f55b1eab48-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:58.623334 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.623291    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d05e78-8832-4213-b976-f7b5b42c0863" path="/var/lib/kubelet/pods/86d05e78-8832-4213-b976-f7b5b42c0863/volumes"
Oct 31 03:46:58.670790 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.670755    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:46:58.689749 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.689711    5415 generic.go:356] "Generic (PLEG): container finished" podID="b5272b1e-7cdc-4d44-a658-50f259988c1e" containerID="bb7255c5d8676971d0e04577e6b9dc3760efb94cf4390fc775f0dbfc7ae23c40" exitCode=0
Oct 31 03:46:58.690007 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.689799    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"
Oct 31 03:46:58.690007 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.689800    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955" event={"ID":"b5272b1e-7cdc-4d44-a658-50f259988c1e","Type":"ContainerDied","Data":"bb7255c5d8676971d0e04577e6b9dc3760efb94cf4390fc775f0dbfc7ae23c40"}
Oct 31 03:46:58.690007 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.689975    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955" event={"ID":"b5272b1e-7cdc-4d44-a658-50f259988c1e","Type":"ContainerDied","Data":"cde885168fcc1aafb7bd0770d6e92a6b09e0bb302494da42acd5d8664411e2a9"}
Oct 31 03:46:58.690184 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.690011    5415 scope.go:122] "RemoveContainer" containerID="bb7255c5d8676971d0e04577e6b9dc3760efb94cf4390fc775f0dbfc7ae23c40"
Oct 31 03:46:58.691718 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.691692    5415 generic.go:356] "Generic (PLEG): container finished" podID="b529469f-b83f-41af-a9a3-35f55b1eab48" containerID="f7404e3d1aee147c2a82dfda365c5181c2ed29bba12b702b8e69a2414e183e01" exitCode=0
Oct 31 03:46:58.691838 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.691755    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"
Oct 31 03:46:58.691838 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.691780    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42" event={"ID":"b529469f-b83f-41af-a9a3-35f55b1eab48","Type":"ContainerDied","Data":"f7404e3d1aee147c2a82dfda365c5181c2ed29bba12b702b8e69a2414e183e01"}
Oct 31 03:46:58.691999 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.691840    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42" event={"ID":"b529469f-b83f-41af-a9a3-35f55b1eab48","Type":"ContainerDied","Data":"6090aa152af40d92c17399b0c44f71e22f8e63d83b142b723b7d4c7c2388d205"}
Oct 31 03:46:58.701277 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.701251    5415 scope.go:122] "RemoveContainer" containerID="bb7255c5d8676971d0e04577e6b9dc3760efb94cf4390fc775f0dbfc7ae23c40"
Oct 31 03:46:58.701639 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:46:58.701612    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7255c5d8676971d0e04577e6b9dc3760efb94cf4390fc775f0dbfc7ae23c40\": container with ID starting with bb7255c5d8676971d0e04577e6b9dc3760efb94cf4390fc775f0dbfc7ae23c40 not found: ID does not exist" containerID="bb7255c5d8676971d0e04577e6b9dc3760efb94cf4390fc775f0dbfc7ae23c40"
Oct 31 03:46:58.701709 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.701653    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7255c5d8676971d0e04577e6b9dc3760efb94cf4390fc775f0dbfc7ae23c40"} err="failed to get container status \"bb7255c5d8676971d0e04577e6b9dc3760efb94cf4390fc775f0dbfc7ae23c40\": rpc error: code = NotFound desc = could not find container \"bb7255c5d8676971d0e04577e6b9dc3760efb94cf4390fc775f0dbfc7ae23c40\": container with ID starting with bb7255c5d8676971d0e04577e6b9dc3760efb94cf4390fc775f0dbfc7ae23c40 not found: ID does not exist"
Oct 31 03:46:58.701709 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.701680    5415 scope.go:122] "RemoveContainer" containerID="f7404e3d1aee147c2a82dfda365c5181c2ed29bba12b702b8e69a2414e183e01"
Oct 31 03:46:58.711042 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.711020    5415 scope.go:122] "RemoveContainer" containerID="f7404e3d1aee147c2a82dfda365c5181c2ed29bba12b702b8e69a2414e183e01"
Oct 31 03:46:58.711380 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:46:58.711351    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7404e3d1aee147c2a82dfda365c5181c2ed29bba12b702b8e69a2414e183e01\": container with ID starting with f7404e3d1aee147c2a82dfda365c5181c2ed29bba12b702b8e69a2414e183e01 not found: ID does not exist" containerID="f7404e3d1aee147c2a82dfda365c5181c2ed29bba12b702b8e69a2414e183e01"
Oct 31 03:46:58.711477 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.711391    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7404e3d1aee147c2a82dfda365c5181c2ed29bba12b702b8e69a2414e183e01"} err="failed to get container status \"f7404e3d1aee147c2a82dfda365c5181c2ed29bba12b702b8e69a2414e183e01\": rpc error: code = NotFound desc = could not find container \"f7404e3d1aee147c2a82dfda365c5181c2ed29bba12b702b8e69a2414e183e01\": container with ID starting with f7404e3d1aee147c2a82dfda365c5181c2ed29bba12b702b8e69a2414e183e01 not found: ID does not exist"
Oct 31 03:46:58.782328 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.782291    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-metrics-certs\") pod \"b5272b1e-7cdc-4d44-a658-50f259988c1e\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") "
Oct 31 03:46:58.782541 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.782355    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5272b1e-7cdc-4d44-a658-50f259988c1e-kube-api-access-p57fv\" (UniqueName: \"kubernetes.io/projected/b5272b1e-7cdc-4d44-a658-50f259988c1e-kube-api-access-p57fv\") pod \"b5272b1e-7cdc-4d44-a658-50f259988c1e\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") "
Oct 31 03:46:58.782541 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.782429    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-maas-api-tls\") pod \"b5272b1e-7cdc-4d44-a658-50f259988c1e\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") "
Oct 31 03:46:58.782541 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.782516    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5272b1e-7cdc-4d44-a658-50f259988c1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5272b1e-7cdc-4d44-a658-50f259988c1e-openshift-service-ca\") pod \"b5272b1e-7cdc-4d44-a658-50f259988c1e\" (UID: \"b5272b1e-7cdc-4d44-a658-50f259988c1e\") "
Oct 31 03:46:58.783296 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.783263    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5272b1e-7cdc-4d44-a658-50f259988c1e-openshift-service-ca" pod "b5272b1e-7cdc-4d44-a658-50f259988c1e" (UID: "b5272b1e-7cdc-4d44-a658-50f259988c1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:58.785774 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.785746    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-metrics-certs" pod "b5272b1e-7cdc-4d44-a658-50f259988c1e" (UID: "b5272b1e-7cdc-4d44-a658-50f259988c1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:46:58.787508 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.787357    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5272b1e-7cdc-4d44-a658-50f259988c1e-kube-api-access-p57fv" pod "b5272b1e-7cdc-4d44-a658-50f259988c1e" (UID: "b5272b1e-7cdc-4d44-a658-50f259988c1e"). InnerVolumeSpecName "kube-api-access-p57fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:58.789067 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.789033    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-maas-api-tls" pod "b5272b1e-7cdc-4d44-a658-50f259988c1e" (UID: "b5272b1e-7cdc-4d44-a658-50f259988c1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:46:58.884160 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.884127    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5272b1e-7cdc-4d44-a658-50f259988c1e-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:58.884160 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.884158    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:58.884375 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.884170    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p57fv\" (UniqueName: \"kubernetes.io/projected/b5272b1e-7cdc-4d44-a658-50f259988c1e-kube-api-access-p57fv\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:58.884375 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.884180    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5272b1e-7cdc-4d44-a658-50f259988c1e-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:58.935226 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.935191    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"]
Oct 31 03:46:58.939177 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:58.939150    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d22078-5c45ddbf8c-56t42"]
Oct 31 03:46:59.104420 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:59.104382    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"]
Oct 31 03:46:59.107596 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:46:59.107564    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a32cdb-74997f57b7-rm955"]
Oct 31 03:47:00.605684 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:00.605647    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5272b1e-7cdc-4d44-a658-50f259988c1e" path="/var/lib/kubelet/pods/b5272b1e-7cdc-4d44-a658-50f259988c1e/volumes"
Oct 31 03:47:00.606105 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:00.606057    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b529469f-b83f-41af-a9a3-35f55b1eab48" path="/var/lib/kubelet/pods/b529469f-b83f-41af-a9a3-35f55b1eab48/volumes"
Oct 31 03:47:16.176260 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:47:16.176222    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:47:28.806950 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:28.806861    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"]
Oct 31 03:47:28.807522 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:28.807128    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4" podUID="f58c5c50-b168-4fc0-9ea1-6fc093b2007e" containerName="maas-api" containerID="cri-o://388f8669c98b3a4bfcbd7fbf6893af4dda3897cbfbfacdedda598c310089b687" gracePeriod=30
Oct 31 03:47:29.092317 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.092280    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:47:29.240337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.239318    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-openshift-service-ca\") pod \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") "
Oct 31 03:47:29.240337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.239452    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-maas-api-tls\") pod \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") "
Oct 31 03:47:29.240337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.239513    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-metrics-certs\") pod \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") "
Oct 31 03:47:29.240337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.239551    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-kube-api-access-mg4vf\" (UniqueName: \"kubernetes.io/projected/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-kube-api-access-mg4vf\") pod \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\" (UID: \"f58c5c50-b168-4fc0-9ea1-6fc093b2007e\") "
Oct 31 03:47:29.242264 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.242228    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-maas-api-tls" pod "f58c5c50-b168-4fc0-9ea1-6fc093b2007e" (UID: "f58c5c50-b168-4fc0-9ea1-6fc093b2007e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:29.243583 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.242828    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-kube-api-access-mg4vf" pod "f58c5c50-b168-4fc0-9ea1-6fc093b2007e" (UID: "f58c5c50-b168-4fc0-9ea1-6fc093b2007e"). InnerVolumeSpecName "kube-api-access-mg4vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:29.245759 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.245714    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-metrics-certs" pod "f58c5c50-b168-4fc0-9ea1-6fc093b2007e" (UID: "f58c5c50-b168-4fc0-9ea1-6fc093b2007e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:29.255267 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.255230    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-openshift-service-ca" pod "f58c5c50-b168-4fc0-9ea1-6fc093b2007e" (UID: "f58c5c50-b168-4fc0-9ea1-6fc093b2007e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:29.340616 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.340580    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg4vf\" (UniqueName: \"kubernetes.io/projected/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-kube-api-access-mg4vf\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:29.340616 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.340618    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:29.340819 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.340631    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:29.340819 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.340641    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f58c5c50-b168-4fc0-9ea1-6fc093b2007e-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:29.870761 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.870723    5415 generic.go:356] "Generic (PLEG): container finished" podID="f58c5c50-b168-4fc0-9ea1-6fc093b2007e" containerID="388f8669c98b3a4bfcbd7fbf6893af4dda3897cbfbfacdedda598c310089b687" exitCode=0
Oct 31 03:47:29.871305 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.870776    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"
Oct 31 03:47:29.871305 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.870803    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4" event={"ID":"f58c5c50-b168-4fc0-9ea1-6fc093b2007e","Type":"ContainerDied","Data":"388f8669c98b3a4bfcbd7fbf6893af4dda3897cbfbfacdedda598c310089b687"}
Oct 31 03:47:29.871305 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.870860    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4" event={"ID":"f58c5c50-b168-4fc0-9ea1-6fc093b2007e","Type":"ContainerDied","Data":"c76753ec8689a27aa2b81522817a30173c26d5f995d0f909c1e34a5a1ebcbb24"}
Oct 31 03:47:29.871305 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.870910    5415 scope.go:122] "RemoveContainer" containerID="388f8669c98b3a4bfcbd7fbf6893af4dda3897cbfbfacdedda598c310089b687"
Oct 31 03:47:29.880779 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.880760    5415 scope.go:122] "RemoveContainer" containerID="388f8669c98b3a4bfcbd7fbf6893af4dda3897cbfbfacdedda598c310089b687"
Oct 31 03:47:29.881121 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:47:29.881100    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388f8669c98b3a4bfcbd7fbf6893af4dda3897cbfbfacdedda598c310089b687\": container with ID starting with 388f8669c98b3a4bfcbd7fbf6893af4dda3897cbfbfacdedda598c310089b687 not found: ID does not exist" containerID="388f8669c98b3a4bfcbd7fbf6893af4dda3897cbfbfacdedda598c310089b687"
Oct 31 03:47:29.881204 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:29.881131    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388f8669c98b3a4bfcbd7fbf6893af4dda3897cbfbfacdedda598c310089b687"} err="failed to get container status \"388f8669c98b3a4bfcbd7fbf6893af4dda3897cbfbfacdedda598c310089b687\": rpc error: code = NotFound desc = could not find container \"388f8669c98b3a4bfcbd7fbf6893af4dda3897cbfbfacdedda598c310089b687\": container with ID starting with 388f8669c98b3a4bfcbd7fbf6893af4dda3897cbfbfacdedda598c310089b687 not found: ID does not exist"
Oct 31 03:47:30.025920 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:30.025860    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"]
Oct 31 03:47:30.072818 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:30.072781    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-03d737-789b8f88c7-sv8h4"]
Oct 31 03:47:30.607008 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:47:30.606970    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58c5c50-b168-4fc0-9ea1-6fc093b2007e" path="/var/lib/kubelet/pods/f58c5c50-b168-4fc0-9ea1-6fc093b2007e/volumes"
Oct 31 03:47:46.175367 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:47:46.175328    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:48:12.040485 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:12.040445    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"]
Oct 31 03:48:16.175233 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:48:16.175145    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:48:17.041747 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:17.041663    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf" podUID="3dd144db-5910-4f9f-bec3-3655053c0152" containerName="payload-processing" containerID="cri-o://6b288065eb113604026b2eb4e4d56d549adad300501a45246ee1d18511d21d70" gracePeriod=25
Oct 31 03:48:17.313392 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:17.313360    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"
Oct 31 03:48:17.382344 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:17.382311    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd144db-5910-4f9f-bec3-3655053c0152-kube-api-access-lm8fq\" (UniqueName: \"kubernetes.io/projected/3dd144db-5910-4f9f-bec3-3655053c0152-kube-api-access-lm8fq\") pod \"3dd144db-5910-4f9f-bec3-3655053c0152\" (UID: \"3dd144db-5910-4f9f-bec3-3655053c0152\") "
Oct 31 03:48:17.382563 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:17.382378    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dd144db-5910-4f9f-bec3-3655053c0152-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd144db-5910-4f9f-bec3-3655053c0152-plugins-config-volume\") pod \"3dd144db-5910-4f9f-bec3-3655053c0152\" (UID: \"3dd144db-5910-4f9f-bec3-3655053c0152\") "
Oct 31 03:48:17.382817 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:17.382796    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dd144db-5910-4f9f-bec3-3655053c0152-plugins-config-volume" pod "3dd144db-5910-4f9f-bec3-3655053c0152" (UID: "3dd144db-5910-4f9f-bec3-3655053c0152"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:17.384518 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:17.384486    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd144db-5910-4f9f-bec3-3655053c0152-kube-api-access-lm8fq" pod "3dd144db-5910-4f9f-bec3-3655053c0152" (UID: "3dd144db-5910-4f9f-bec3-3655053c0152"). InnerVolumeSpecName "kube-api-access-lm8fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:17.484108 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:17.484067    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm8fq\" (UniqueName: \"kubernetes.io/projected/3dd144db-5910-4f9f-bec3-3655053c0152-kube-api-access-lm8fq\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:17.484108 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:17.484100    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd144db-5910-4f9f-bec3-3655053c0152-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:18.133484 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:18.133446    5415 generic.go:356] "Generic (PLEG): container finished" podID="3dd144db-5910-4f9f-bec3-3655053c0152" containerID="6b288065eb113604026b2eb4e4d56d549adad300501a45246ee1d18511d21d70" exitCode=0
Oct 31 03:48:18.133701 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:18.133494    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf" event={"ID":"3dd144db-5910-4f9f-bec3-3655053c0152","Type":"ContainerDied","Data":"6b288065eb113604026b2eb4e4d56d549adad300501a45246ee1d18511d21d70"}
Oct 31 03:48:18.133701 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:18.133503    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"
Oct 31 03:48:18.133701 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:18.133536    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf" event={"ID":"3dd144db-5910-4f9f-bec3-3655053c0152","Type":"ContainerDied","Data":"6fa99d7bc8d05e706a85df5dea5301a081ccffd0445dc4ff741929bf3a3805a5"}
Oct 31 03:48:18.133701 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:18.133557    5415 scope.go:122] "RemoveContainer" containerID="6b288065eb113604026b2eb4e4d56d549adad300501a45246ee1d18511d21d70"
Oct 31 03:48:18.143954 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:18.143930    5415 scope.go:122] "RemoveContainer" containerID="6b288065eb113604026b2eb4e4d56d549adad300501a45246ee1d18511d21d70"
Oct 31 03:48:18.144302 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:48:18.144275    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b288065eb113604026b2eb4e4d56d549adad300501a45246ee1d18511d21d70\": container with ID starting with 6b288065eb113604026b2eb4e4d56d549adad300501a45246ee1d18511d21d70 not found: ID does not exist" containerID="6b288065eb113604026b2eb4e4d56d549adad300501a45246ee1d18511d21d70"
Oct 31 03:48:18.144363 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:18.144314    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b288065eb113604026b2eb4e4d56d549adad300501a45246ee1d18511d21d70"} err="failed to get container status \"6b288065eb113604026b2eb4e4d56d549adad300501a45246ee1d18511d21d70\": rpc error: code = NotFound desc = could not find container \"6b288065eb113604026b2eb4e4d56d549adad300501a45246ee1d18511d21d70\": container with ID starting with 6b288065eb113604026b2eb4e4d56d549adad300501a45246ee1d18511d21d70 not found: ID does not exist"
Oct 31 03:48:18.288713 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:18.288671    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"]
Oct 31 03:48:18.335796 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:18.335764    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dab9ee-6d996d455-8lcdf"]
Oct 31 03:48:18.605572 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:18.605540    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dd144db-5910-4f9f-bec3-3655053c0152" path="/var/lib/kubelet/pods/3dd144db-5910-4f9f-bec3-3655053c0152/volumes"
Oct 31 03:48:20.338577 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.338543    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"]
Oct 31 03:48:20.339042 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339028    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8c780d3a-6662-4c40-b816-3cf1f6da41ff" containerName="payload-processing"
Oct 31 03:48:20.339090 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339043    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c780d3a-6662-4c40-b816-3cf1f6da41ff" containerName="payload-processing"
Oct 31 03:48:20.339090 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339052    5415 cpu_manager.go:414] "containerMap: removing container" podUID="587c546c-eb63-434b-aeea-45527d431c43" containerName="cleanup"
Oct 31 03:48:20.339090 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339057    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587c546c-eb63-434b-aeea-45527d431c43" containerName="cleanup"
Oct 31 03:48:20.339090 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339065    5415 cpu_manager.go:414] "containerMap: removing container" podUID="b5272b1e-7cdc-4d44-a658-50f259988c1e" containerName="maas-api"
Oct 31 03:48:20.339090 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339069    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5272b1e-7cdc-4d44-a658-50f259988c1e" containerName="maas-api"
Oct 31 03:48:20.339090 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339076    5415 cpu_manager.go:414] "containerMap: removing container" podUID="7fffc7b3-173d-4747-8362-aaeb3e57972b" containerName="cleanup"
Oct 31 03:48:20.339090 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339081    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fffc7b3-173d-4747-8362-aaeb3e57972b" containerName="cleanup"
Oct 31 03:48:20.339090 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339090    5415 cpu_manager.go:414] "containerMap: removing container" podUID="156d2f57-24b3-43a4-9a06-3bfa3a7f4df6" containerName="maas-api"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339096    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156d2f57-24b3-43a4-9a06-3bfa3a7f4df6" containerName="maas-api"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339110    5415 cpu_manager.go:414] "containerMap: removing container" podUID="da356000-9c44-4fa1-9d31-0dd6e65145f4" containerName="payload-pre-processing"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339115    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da356000-9c44-4fa1-9d31-0dd6e65145f4" containerName="payload-pre-processing"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339123    5415 cpu_manager.go:414] "containerMap: removing container" podUID="587c546c-eb63-434b-aeea-45527d431c43" containerName="cleanup"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339127    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587c546c-eb63-434b-aeea-45527d431c43" containerName="cleanup"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339135    5415 cpu_manager.go:414] "containerMap: removing container" podUID="3dd144db-5910-4f9f-bec3-3655053c0152" containerName="payload-processing"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339139    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd144db-5910-4f9f-bec3-3655053c0152" containerName="payload-processing"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339145    5415 cpu_manager.go:414] "containerMap: removing container" podUID="587c546c-eb63-434b-aeea-45527d431c43" containerName="cleanup"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339150    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587c546c-eb63-434b-aeea-45527d431c43" containerName="cleanup"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339160    5415 cpu_manager.go:414] "containerMap: removing container" podUID="87788dd8-2aa4-4b65-80f2-8116be88a903" containerName="main"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339164    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87788dd8-2aa4-4b65-80f2-8116be88a903" containerName="main"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339170    5415 cpu_manager.go:414] "containerMap: removing container" podUID="b529469f-b83f-41af-a9a3-35f55b1eab48" containerName="payload-processing"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339175    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b529469f-b83f-41af-a9a3-35f55b1eab48" containerName="payload-processing"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339181    5415 cpu_manager.go:414] "containerMap: removing container" podUID="86d05e78-8832-4213-b976-f7b5b42c0863" containerName="maas-api"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339187    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d05e78-8832-4213-b976-f7b5b42c0863" containerName="maas-api"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339196    5415 cpu_manager.go:414] "containerMap: removing container" podUID="7fffc7b3-173d-4747-8362-aaeb3e57972b" containerName="cleanup"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339200    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fffc7b3-173d-4747-8362-aaeb3e57972b" containerName="cleanup"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339207    5415 cpu_manager.go:414] "containerMap: removing container" podUID="f58c5c50-b168-4fc0-9ea1-6fc093b2007e" containerName="maas-api"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339211    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58c5c50-b168-4fc0-9ea1-6fc093b2007e" containerName="maas-api"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339281    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58c5c50-b168-4fc0-9ea1-6fc093b2007e" containerName="maas-api"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339292    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c780d3a-6662-4c40-b816-3cf1f6da41ff" containerName="payload-processing"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339298    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="587c546c-eb63-434b-aeea-45527d431c43" containerName="cleanup"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339303    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5272b1e-7cdc-4d44-a658-50f259988c1e" containerName="maas-api"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339308    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fffc7b3-173d-4747-8362-aaeb3e57972b" containerName="cleanup"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339313    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="156d2f57-24b3-43a4-9a06-3bfa3a7f4df6" containerName="maas-api"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339320    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="da356000-9c44-4fa1-9d31-0dd6e65145f4" containerName="payload-pre-processing"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339326    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd144db-5910-4f9f-bec3-3655053c0152" containerName="payload-processing"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339331    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="587c546c-eb63-434b-aeea-45527d431c43" containerName="cleanup"
Oct 31 03:48:20.339327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339338    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="87788dd8-2aa4-4b65-80f2-8116be88a903" containerName="main"
Oct 31 03:48:20.340208 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339343    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="b529469f-b83f-41af-a9a3-35f55b1eab48" containerName="payload-processing"
Oct 31 03:48:20.340208 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339349    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d05e78-8832-4213-b976-f7b5b42c0863" containerName="maas-api"
Oct 31 03:48:20.340208 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.339354    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fffc7b3-173d-4747-8362-aaeb3e57972b" containerName="cleanup"
Oct 31 03:48:20.344071 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.344046    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:20.442986 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.442951    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ca15df-serving-cert\""
Oct 31 03:48:20.444641 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.444617    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"]
Oct 31 03:48:20.517352 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.517316    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa81a6e-d0b4-43b6-8bba-87d56abba045-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:20.517576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.517391    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smcmm\" (UniqueName: \"kubernetes.io/projected/8fa81a6e-d0b4-43b6-8bba-87d56abba045-kube-api-access-smcmm\") pod \"maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:20.517576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.517539    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-metrics-certs\") pod \"maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:20.517689 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.517630    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:20.619140 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.619047    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smcmm\" (UniqueName: \"kubernetes.io/projected/8fa81a6e-d0b4-43b6-8bba-87d56abba045-kube-api-access-smcmm\") pod \"maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:20.619140 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.619121    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-metrics-certs\") pod \"maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:20.619379 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.619170    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:20.619379 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.619280    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa81a6e-d0b4-43b6-8bba-87d56abba045-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:20.619379 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:48:20.619311    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-serving-cert: secret "maas-api-e2e-ipp-b-ca15df-serving-cert" not found
Oct 31 03:48:20.619494 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:48:20.619384    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-maas-api-tls podName:8fa81a6e-d0b4-43b6-8bba-87d56abba045 nodeName:}" failed. No retries permitted until 2026-10-31 03:48:21.119362668 +0000 UTC m=+2442.679842391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-maas-api-tls") pod "maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc" (UID: "8fa81a6e-d0b4-43b6-8bba-87d56abba045") : secret "maas-api-e2e-ipp-b-ca15df-serving-cert" not found
Oct 31 03:48:20.620021 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.619998    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa81a6e-d0b4-43b6-8bba-87d56abba045-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:20.627885 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.627855    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-metrics-certs\") pod \"maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:20.674088 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:20.674049    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smcmm\" (UniqueName: \"kubernetes.io/projected/8fa81a6e-d0b4-43b6-8bba-87d56abba045-kube-api-access-smcmm\") pod \"maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:21.126966 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:21.126919    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:21.129392 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:21.129359    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:21.257617 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:21.257582    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:21.631293 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:21.631258    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"]
Oct 31 03:48:21.634632 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:48:21.634599    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa81a6e_d0b4_43b6_8bba_87d56abba045.slice/crio-a0ccd6571c6fec77f53b3984f6cc120b30d476c7c1e68e0e9d38ea9df1a0f905 WatchSource:0}: Error finding container a0ccd6571c6fec77f53b3984f6cc120b30d476c7c1e68e0e9d38ea9df1a0f905: Status 404 returned error can't find the container with id a0ccd6571c6fec77f53b3984f6cc120b30d476c7c1e68e0e9d38ea9df1a0f905
Oct 31 03:48:22.159323 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:22.159281    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc" event={"ID":"8fa81a6e-d0b4-43b6-8bba-87d56abba045","Type":"ContainerStarted","Data":"a0ccd6571c6fec77f53b3984f6cc120b30d476c7c1e68e0e9d38ea9df1a0f905"}
Oct 31 03:48:23.166609 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:23.166570    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc" event={"ID":"8fa81a6e-d0b4-43b6-8bba-87d56abba045","Type":"ContainerStarted","Data":"42b9112d1a81b956f6ba5c5c53d043f2b3194d3c869bc0fba0bd2cf98d46491e"}
Oct 31 03:48:23.167052 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:23.166693    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:23.277332 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:23.277276    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc" podStartSLOduration=2.331738281 podStartE2EDuration="3.277261788s" totalImagesPullingTime="945.523507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:23.274766949 +0000 UTC m=+2444.835246681" watchObservedRunningTime="2026-10-31 03:48:23.277261788 +0000 UTC m=+2444.837741516"
Oct 31 03:48:29.177729 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:29.177698    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:48:40.498359 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.498322    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"]
Oct 31 03:48:40.498986 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.498966    5415 cpu_manager.go:414] "containerMap: removing container" podUID="7fffc7b3-173d-4747-8362-aaeb3e57972b" containerName="cleanup"
Oct 31 03:48:40.498986 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.498986    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fffc7b3-173d-4747-8362-aaeb3e57972b" containerName="cleanup"
Oct 31 03:48:40.499114 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.499103    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="587c546c-eb63-434b-aeea-45527d431c43" containerName="cleanup"
Oct 31 03:48:40.499149 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.499116    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fffc7b3-173d-4747-8362-aaeb3e57972b" containerName="cleanup"
Oct 31 03:48:40.506439 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.506414    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.559247 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.559205    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9360eb-openshift-default-dockercfg-lsbz7\""
Oct 31 03:48:40.561752 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.561722    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"]
Oct 31 03:48:40.655031 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.654995    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.655031 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.655033    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjzpm\" (UniqueName: \"kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-kube-api-access-xjzpm\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.655284 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.655051    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.655284 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.655229    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.655376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.655313    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.655376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.655338    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.655376 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.655373    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.655505 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.655421    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.655505 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.655493    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.756315 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.756202    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.756315 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.756305    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.756567 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.756377    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.756567 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.756407    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.756567 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.756452    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.756567 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.756499    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.756739 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.756581    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.756739 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.756655    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.756739 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.756696    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.756739 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.756730    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjzpm\" (UniqueName: \"kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-kube-api-access-xjzpm\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.756945 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.756852    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.757196 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.757148    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.757339 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.757285    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.757497 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.757478    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.759242 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.759214    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.759348 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.759256    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.811494 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.811451    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.811794 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.811768    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjzpm\" (UniqueName: \"kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-kube-api-access-xjzpm\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:40.822539 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:40.822491    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:41.243669 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:41.243644    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"]
Oct 31 03:48:41.246328 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:48:41.246296    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba7e0d3_2990_4db0_a832_b2d40e8fa461.slice/crio-55eb208b243a103e857609bd43a2517e2f25e9995e4bd3f306076eff0de2d303 WatchSource:0}: Error finding container 55eb208b243a103e857609bd43a2517e2f25e9995e4bd3f306076eff0de2d303: Status 404 returned error can't find the container with id 55eb208b243a103e857609bd43a2517e2f25e9995e4bd3f306076eff0de2d303
Oct 31 03:48:41.248702 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:41.248664    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:48:41.248812 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:41.248739    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:48:41.248812 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:41.248769    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:48:41.265720 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:41.265688    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm" event={"ID":"fba7e0d3-2990-4db0-a832-b2d40e8fa461","Type":"ContainerStarted","Data":"55eb208b243a103e857609bd43a2517e2f25e9995e4bd3f306076eff0de2d303"}
Oct 31 03:48:42.286341 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:42.286303    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm" event={"ID":"fba7e0d3-2990-4db0-a832-b2d40e8fa461","Type":"ContainerStarted","Data":"01a246a3370a9f8912b8e7d5dc33a55ade51ce5a4a2f7e6b25159f2620943899"}
Oct 31 03:48:42.445332 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:42.445280    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm" podStartSLOduration=2.445265049 podStartE2EDuration="2.445265049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:42.444372825 +0000 UTC m=+2464.004852556" watchObservedRunningTime="2026-10-31 03:48:42.445265049 +0000 UTC m=+2464.005744779"
Oct 31 03:48:42.823477 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:42.823443    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:42.825352 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:42.825321    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:48:42.825504 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:42.825373    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm" podUID="fba7e0d3-2990-4db0-a832-b2d40e8fa461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:48:43.822846 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:43.822809    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:48:43.823297 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:43.822869    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm" podUID="fba7e0d3-2990-4db0-a832-b2d40e8fa461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:48:44.822910 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:44.822846    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:48:44.823424 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:44.822922    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm" podUID="fba7e0d3-2990-4db0-a832-b2d40e8fa461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:48:45.823283 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:45.823247    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:48:45.823714 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:45.823332    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm" podUID="fba7e0d3-2990-4db0-a832-b2d40e8fa461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:48:46.196895 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:48:46.196845    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:48:46.823156 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:46.823117    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:48:46.823346 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:46.823185    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm" podUID="fba7e0d3-2990-4db0-a832-b2d40e8fa461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:48:47.822834 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:47.822797    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:48:47.823038 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:47.822858    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm" podUID="fba7e0d3-2990-4db0-a832-b2d40e8fa461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:48:49.127102 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:49.127067    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:49.127533 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:49.127484    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:48:49.128530 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:48:49.128509    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 03:49:16.176451 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:49:16.176409    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:49:40.966855 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:40.966799    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"]
Oct 31 03:49:40.970672 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:40.970652    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.036661 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.036623    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.036850 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.036676    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.036850 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.036705    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19da59fb-854b-4e33-b635-c68551bf4179-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.036850 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.036800    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.036850 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.036843    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19da59fb-854b-4e33-b635-c68551bf4179-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.037026 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.036869    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.037026 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.036938    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5nj\" (UniqueName: \"kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-kube-api-access-7b5nj\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.037026 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.036963    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.037026 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.036983    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.074276 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.074238    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"]
Oct 31 03:49:41.137759 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.137723    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.138011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.137771    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.138011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.137793    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19da59fb-854b-4e33-b635-c68551bf4179-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.138011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.137827    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.138011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.137859    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19da59fb-854b-4e33-b635-c68551bf4179-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.138011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.137911    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.138011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.137943    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5nj\" (UniqueName: \"kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-kube-api-access-7b5nj\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.138011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.137980    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.138011 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.138002    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.138412 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.138278    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.138412 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.138334    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.138412 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.138399    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.138589 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.138489    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.138729 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.138706    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19da59fb-854b-4e33-b635-c68551bf4179-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.140372 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.140343    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.140596 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.140580    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19da59fb-854b-4e33-b635-c68551bf4179-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.192245 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.192210    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.192397 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.192297    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5nj\" (UniqueName: \"kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-kube-api-access-7b5nj\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.283434 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.283345    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:41.703823 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.703785    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"]
Oct 31 03:49:41.742900 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:49:41.742851    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19da59fb_854b_4e33_b635_c68551bf4179.slice/crio-e12c8f5ecf153cf5a1bfb2c3c75c47a613c59a7ea1dd20f8cf267b5a14eac173 WatchSource:0}: Error finding container e12c8f5ecf153cf5a1bfb2c3c75c47a613c59a7ea1dd20f8cf267b5a14eac173: Status 404 returned error can't find the container with id e12c8f5ecf153cf5a1bfb2c3c75c47a613c59a7ea1dd20f8cf267b5a14eac173
Oct 31 03:49:41.745260 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.745225    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:49:41.745343 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.745307    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:49:41.745343 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:41.745337    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:49:42.621699 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:42.621658    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h" event={"ID":"19da59fb-854b-4e33-b635-c68551bf4179","Type":"ContainerStarted","Data":"4ce133667629a5036058142c7a207e5627c0e59e103a6fcb46aa5a35c59d40a7"}
Oct 31 03:49:42.622157 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:42.621722    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h" event={"ID":"19da59fb-854b-4e33-b635-c68551bf4179","Type":"ContainerStarted","Data":"e12c8f5ecf153cf5a1bfb2c3c75c47a613c59a7ea1dd20f8cf267b5a14eac173"}
Oct 31 03:49:42.784696 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:42.784627    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h" podStartSLOduration=2.78460588 podStartE2EDuration="2.78460588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:42.78173543 +0000 UTC m=+2524.342215161" watchObservedRunningTime="2026-10-31 03:49:42.78460588 +0000 UTC m=+2524.345085609"
Oct 31 03:49:43.283499 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:43.283467    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:43.284935 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:43.284907    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:49:43.285072 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:43.284956    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h" podUID="19da59fb-854b-4e33-b635-c68551bf4179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:49:44.284368 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:44.284326    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:49:44.284897 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:44.284401    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h" podUID="19da59fb-854b-4e33-b635-c68551bf4179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:49:45.284673 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:45.284587    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:49:45.285109 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:45.284667    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h" podUID="19da59fb-854b-4e33-b635-c68551bf4179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:49:46.200126 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:49:46.200085    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:49:46.334114 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:46.334072    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:49:46.334732 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:46.334696    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h" podUID="19da59fb-854b-4e33-b635-c68551bf4179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:49:47.284765 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:47.284719    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:49:47.285086 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:47.284785    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h" podUID="19da59fb-854b-4e33-b635-c68551bf4179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:49:48.284529 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:48.284491    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:49:48.284983 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:48.284547    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h" podUID="19da59fb-854b-4e33-b635-c68551bf4179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:49:49.289022 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:49.288976    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:49.289586 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:49.289546    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:49.290409 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:49.290387    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:49:56.106817 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.106768    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"]
Oct 31 03:49:56.110844 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.110826    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.210110 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.210065    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"]
Oct 31 03:49:56.281604 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.281568    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.281791 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.281625    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.281791 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.281683    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.281791 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.281724    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.281791 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.281776    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.281958 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.281833    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.281958 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.281940    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.282021 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.281984    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.282064 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.282051    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnhdb\" (UniqueName: \"kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-kube-api-access-jnhdb\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.383118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.383039    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.383325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.383196    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.383325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.383224    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.383325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.383258    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.383325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.383288    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnhdb\" (UniqueName: \"kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-kube-api-access-jnhdb\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.383527 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.383351    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.383527 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.383409    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.383527 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.383441    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.383527 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.383462    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.383765 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.383656    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.383824 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.383784    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.384013 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.383987    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.384239 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.384207    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.384336 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.384317    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.386052 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.386031    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.386188 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.386165    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.443056 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.443027    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.443211 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.443129    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnhdb\" (UniqueName: \"kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-kube-api-access-jnhdb\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:56.724428 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:56.724335    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:57.142255 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:57.142220    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"]
Oct 31 03:49:57.144270 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:49:57.144242    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64a42a6_ace4_42e2_bf65_f834e9c4c263.slice/crio-11f0bb2d608e81825bf9f09fb988dc6c4169998778fc2f96244fe3124ddbdb9e WatchSource:0}: Error finding container 11f0bb2d608e81825bf9f09fb988dc6c4169998778fc2f96244fe3124ddbdb9e: Status 404 returned error can't find the container with id 11f0bb2d608e81825bf9f09fb988dc6c4169998778fc2f96244fe3124ddbdb9e
Oct 31 03:49:57.146766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:57.146735    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:49:57.146871 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:57.146803    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:49:57.146871 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:57.146838    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:49:57.713144 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:57.713108    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf" event={"ID":"c64a42a6-ace4-42e2-bf65-f834e9c4c263","Type":"ContainerStarted","Data":"432d02ac44ce018baae7501663149cd9990bbb6c0c3e96a75015fee4b4f0f38e"}
Oct 31 03:49:57.713325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:57.713164    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf" event={"ID":"c64a42a6-ace4-42e2-bf65-f834e9c4c263","Type":"ContainerStarted","Data":"11f0bb2d608e81825bf9f09fb988dc6c4169998778fc2f96244fe3124ddbdb9e"}
Oct 31 03:49:57.725455 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:57.725427    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:49:57.824429 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:57.824372    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf" podStartSLOduration=1.824354483 podStartE2EDuration="1.824354483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:57.822209324 +0000 UTC m=+2539.382689081" watchObservedRunningTime="2026-10-31 03:49:57.824354483 +0000 UTC m=+2539.384834212"
Oct 31 03:49:58.726910 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:58.726855    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:49:58.727302 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:58.726928    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf" podUID="c64a42a6-ace4-42e2-bf65-f834e9c4c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 03:49:59.724712 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:59.724676    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:49:59.724905 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:49:59.724742    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf" podUID="c64a42a6-ace4-42e2-bf65-f834e9c4c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 03:50:00.725578 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:00.725543    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:50:00.725986 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:00.725603    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf" podUID="c64a42a6-ace4-42e2-bf65-f834e9c4c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 03:50:01.725484 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:01.725446    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:50:01.725681 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:01.725505    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf" podUID="c64a42a6-ace4-42e2-bf65-f834e9c4c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 03:50:02.724849 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:02.724809    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:50:02.725079 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:02.724924    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf" podUID="c64a42a6-ace4-42e2-bf65-f834e9c4c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 03:50:03.725284 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:03.725250    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:50:03.725696 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:03.725316    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf" podUID="c64a42a6-ace4-42e2-bf65-f834e9c4c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 03:50:04.729267 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:04.729233    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:50:04.729765 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:04.729632    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:50:04.730449 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:04.730428    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:50:16.269555 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:50:16.269510    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:50:44.772572 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.772532    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"]
Oct 31 03:50:44.779557 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.779518    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:44.818912 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.818843    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"]
Oct 31 03:50:44.823180 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.823148    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:44.866833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.866793    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"]
Oct 31 03:50:44.893560 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.893525    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"]
Oct 31 03:50:44.932989 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.932941    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:44.932989 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.932987    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:44.933307 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.933060    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:44.933307 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.933108    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:44.933430 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.933297    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:44.933430 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.933358    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:44.933430 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.933396    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:44.933582 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.933472    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:44.933582 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.933503    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:44.933582 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.933536    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:44.933735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.933580    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:44.933735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.933629    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kld4n\" (UniqueName: \"kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-kube-api-access-kld4n\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:44.933735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.933674    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4cf674-d047-4b52-9081-1d89348f2546-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:44.933928 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.933758    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4cf674-d047-4b52-9081-1d89348f2546-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:44.934037 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.934005    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:44.934141 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.934120    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:44.934271 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.934241    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:44.934392 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:44.934297    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dww24\" (UniqueName: \"kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-kube-api-access-dww24\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.035065 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.034986    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.035065 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035028    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.035065 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035067    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.035337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035092    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dww24\" (UniqueName: \"kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-kube-api-access-dww24\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.035337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035111    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.035337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035131    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.035337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035154    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.035337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035191    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.035337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035231    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.035337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035262    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.035337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035287    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.035337 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035336    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.035788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035363    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.035788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035480    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.035788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035525    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.035788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035564    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.035788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035564    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.035788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035601    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.035788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035571    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kld4n\" (UniqueName: \"kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-kube-api-access-kld4n\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.035788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035661    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4cf674-d047-4b52-9081-1d89348f2546-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.035788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035733    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4cf674-d047-4b52-9081-1d89348f2546-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.035788 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035781    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.036325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035810    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.036325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035887    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.036325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.035892    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.036325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.036117    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.036585 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.036559    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4cf674-d047-4b52-9081-1d89348f2546-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.036862 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.036840    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.037915 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.037863    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.038069 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.038047    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.038180 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.038108    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.038485 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.038465    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4cf674-d047-4b52-9081-1d89348f2546-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.090450 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.090407    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.090636 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.090476    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.090636 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.090585    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dww24\" (UniqueName: \"kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-kube-api-access-dww24\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.090636 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.090629    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kld4n\" (UniqueName: \"kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-kube-api-access-kld4n\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.095458 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.095437    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:45.139561 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.139519    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:45.514581 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.514557    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"]
Oct 31 03:50:45.516662 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:50:45.516624    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf707fded_4ede_4eb3_99a2_9c16ccb99f50.slice/crio-48495d534307354706408f5b3c6debea121560354fbb232665fc7f95666d0bc0 WatchSource:0}: Error finding container 48495d534307354706408f5b3c6debea121560354fbb232665fc7f95666d0bc0: Status 404 returned error can't find the container with id 48495d534307354706408f5b3c6debea121560354fbb232665fc7f95666d0bc0
Oct 31 03:50:45.519364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.519333    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:50:45.519459 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.519400    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:50:45.519459 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.519428    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:50:45.564946 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.564916    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"]
Oct 31 03:50:45.566922 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:50:45.566894    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4cf674_d047_4b52_9081_1d89348f2546.slice/crio-e5949cada8d713f1c2d69d9b57dd9002bea3c51c791029b6120e46b9809eae1a WatchSource:0}: Error finding container e5949cada8d713f1c2d69d9b57dd9002bea3c51c791029b6120e46b9809eae1a: Status 404 returned error can't find the container with id e5949cada8d713f1c2d69d9b57dd9002bea3c51c791029b6120e46b9809eae1a
Oct 31 03:50:45.569821 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.569787    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:50:45.569943 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.569868    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:50:45.569943 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:45.569929    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 03:50:46.023941 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:46.023870    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" event={"ID":"dc4cf674-d047-4b52-9081-1d89348f2546","Type":"ContainerStarted","Data":"4fd3cfee241df65d26b1d0f258c5fd3e58105132f0083372cfa20a36b96f8e0d"}
Oct 31 03:50:46.024447 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:46.023967    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" event={"ID":"dc4cf674-d047-4b52-9081-1d89348f2546","Type":"ContainerStarted","Data":"e5949cada8d713f1c2d69d9b57dd9002bea3c51c791029b6120e46b9809eae1a"}
Oct 31 03:50:46.025735 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:46.025709    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" event={"ID":"f707fded-4ede-4eb3-99a2-9c16ccb99f50","Type":"ContainerStarted","Data":"c9ef1ca18632d38edb7da8649c0275aa498401f2e8cedf56c258c054c3b833cc"}
Oct 31 03:50:46.025890 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:46.025751    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" event={"ID":"f707fded-4ede-4eb3-99a2-9c16ccb99f50","Type":"ContainerStarted","Data":"48495d534307354706408f5b3c6debea121560354fbb232665fc7f95666d0bc0"}
Oct 31 03:50:46.096306 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:46.096259    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:46.097776 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:46.097752    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:50:46.097931 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:46.097860    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" podUID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 03:50:46.139793 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:46.139716    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:46.141455 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:46.141428    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:50:46.141589 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:46.141483    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" podUID="dc4cf674-d047-4b52-9081-1d89348f2546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 03:50:46.143120 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:46.143070    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" podStartSLOduration=2.143056964 podStartE2EDuration="2.143056964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:46.140417924 +0000 UTC m=+2587.700897655" watchObservedRunningTime="2026-10-31 03:50:46.143056964 +0000 UTC m=+2587.703536694"
Oct 31 03:50:46.197717 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:50:46.197682    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:50:46.281548 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:46.281394    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" podStartSLOduration=2.281369925 podStartE2EDuration="2.281369925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:46.265185654 +0000 UTC m=+2587.825665421" watchObservedRunningTime="2026-10-31 03:50:46.281369925 +0000 UTC m=+2587.841849654"
Oct 31 03:50:47.096535 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:47.096484    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:50:47.097008 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:47.096564    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" podUID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 03:50:47.140903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:47.140853    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:50:47.141143 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:47.141120    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" podUID="dc4cf674-d047-4b52-9081-1d89348f2546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 03:50:48.096904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:48.096321    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:50:48.096904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:48.096383    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" podUID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 03:50:48.140163 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:48.140127    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:50:48.140342 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:48.140180    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" podUID="dc4cf674-d047-4b52-9081-1d89348f2546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 03:50:49.099923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:49.096349    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:50:49.099923 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:49.096435    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" podUID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 03:50:49.140644 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:49.140607    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:50:49.140816 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:49.140689    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" podUID="dc4cf674-d047-4b52-9081-1d89348f2546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 03:50:50.102904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:50.097053    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:50:50.102904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:50.097152    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" podUID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 03:50:50.142330 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:50.140284    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:50:50.142330 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:50.140377    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" podUID="dc4cf674-d047-4b52-9081-1d89348f2546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 03:50:51.096814 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:51.096340    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:50:51.096814 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:51.096412    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" podUID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 03:50:51.140746 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:51.140705    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:50:51.141191 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:51.140765    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" podUID="dc4cf674-d047-4b52-9081-1d89348f2546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 03:50:52.096404 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:52.096359    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:50:52.096628 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:52.096433    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" podUID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 03:50:52.141101 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:52.141049    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:50:52.141649 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:52.141122    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" podUID="dc4cf674-d047-4b52-9081-1d89348f2546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 03:50:53.103305 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:53.103270    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:53.103683 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:53.103665    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:53.104369 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:53.104347    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:50:53.177775 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:53.177717    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:53.178300 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:53.178215    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:50:53.179061 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:50:53.179038    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 03:51:16.174819 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:51:16.174721    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:51:27.946455 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:27.946420    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"]
Oct 31 03:51:27.951662 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:27.951639    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.001331 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.001295    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-2339ed-serving-cert\""
Oct 31 03:51:28.019831 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.019787    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.020043 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.019931    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad27d6d-957f-4a3b-bcef-5a1bab901463-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.020121 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.020049    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.020186 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.020123    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7245g\" (UniqueName: \"kubernetes.io/projected/2ad27d6d-957f-4a3b-bcef-5a1bab901463-kube-api-access-7245g\") pod \"maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.050673 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.050629    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"]
Oct 31 03:51:28.121345 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.121298    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad27d6d-957f-4a3b-bcef-5a1bab901463-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.121567 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.121428    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.121567 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.121539    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7245g\" (UniqueName: \"kubernetes.io/projected/2ad27d6d-957f-4a3b-bcef-5a1bab901463-kube-api-access-7245g\") pod \"maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.121691 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:51:28.121587    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-serving-cert: secret "maas-api-e2e-shared-a-w2-2339ed-serving-cert" not found
Oct 31 03:51:28.121691 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.121610    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.121691 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:51:28.121678    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-maas-api-tls podName:2ad27d6d-957f-4a3b-bcef-5a1bab901463 nodeName:}" failed. No retries permitted until 2026-10-31 03:51:28.621653211 +0000 UTC m=+2630.182132944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-maas-api-tls") pod "maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g" (UID: "2ad27d6d-957f-4a3b-bcef-5a1bab901463") : secret "maas-api-e2e-shared-a-w2-2339ed-serving-cert" not found
Oct 31 03:51:28.122099 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.122076    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad27d6d-957f-4a3b-bcef-5a1bab901463-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.124161 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.124139    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.176444 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.176405    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7245g\" (UniqueName: \"kubernetes.io/projected/2ad27d6d-957f-4a3b-bcef-5a1bab901463-kube-api-access-7245g\") pod \"maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.628935 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.628867    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.631458 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.631435    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:28.865574 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:28.865524    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:29.236525 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:29.236496    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"]
Oct 31 03:51:29.239538 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:51:29.239502    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad27d6d_957f_4a3b_bcef_5a1bab901463.slice/crio-fc82b6e38cc6f003019c8625877dc0137a30cc0b831989cea53d1b8ac88a1680 WatchSource:0}: Error finding container fc82b6e38cc6f003019c8625877dc0137a30cc0b831989cea53d1b8ac88a1680: Status 404 returned error can't find the container with id fc82b6e38cc6f003019c8625877dc0137a30cc0b831989cea53d1b8ac88a1680
Oct 31 03:51:29.241058 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:29.241037    5415 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:51:29.353536 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:29.353499    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g" event={"ID":"2ad27d6d-957f-4a3b-bcef-5a1bab901463","Type":"ContainerStarted","Data":"fc82b6e38cc6f003019c8625877dc0137a30cc0b831989cea53d1b8ac88a1680"}
Oct 31 03:51:31.366494 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:31.366458    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g" event={"ID":"2ad27d6d-957f-4a3b-bcef-5a1bab901463","Type":"ContainerStarted","Data":"8396114e460bb74c400becf1eeb25839051fb5db24c19d7a62f0b5d5b20623a3"}
Oct 31 03:51:31.366945 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:31.366544    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:31.476320 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:31.476267    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g" podStartSLOduration=3.420460549 podStartE2EDuration="4.476251127s" totalImagesPullingTime="1.055790578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:31.474524725 +0000 UTC m=+2633.035004456" watchObservedRunningTime="2026-10-31 03:51:31.476251127 +0000 UTC m=+2633.036730856"
Oct 31 03:51:32.527606 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:32.527571    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"]
Oct 31 03:51:32.528088 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:32.527805    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf" podUID="9b94b81c-0c49-43ea-8a9d-d656802f1ade" containerName="maas-api" containerID="cri-o://ce104525b06f97e3ae255bb18d6812aec70a457217cbda44e72552cced99165c" gracePeriod=30
Oct 31 03:51:32.840429 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:32.840403    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:51:32.990597 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:32.990560    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b94b81c-0c49-43ea-8a9d-d656802f1ade-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b94b81c-0c49-43ea-8a9d-d656802f1ade-openshift-service-ca\") pod \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") "
Oct 31 03:51:32.990819 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:32.990765    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-metrics-certs\") pod \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") "
Oct 31 03:51:32.990925 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:32.990827    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-maas-api-tls\") pod \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") "
Oct 31 03:51:32.991013 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:32.990990    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b94b81c-0c49-43ea-8a9d-d656802f1ade-kube-api-access-8lncp\" (UniqueName: \"kubernetes.io/projected/9b94b81c-0c49-43ea-8a9d-d656802f1ade-kube-api-access-8lncp\") pod \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\" (UID: \"9b94b81c-0c49-43ea-8a9d-d656802f1ade\") "
Oct 31 03:51:32.991663 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:32.991635    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b94b81c-0c49-43ea-8a9d-d656802f1ade-openshift-service-ca" pod "9b94b81c-0c49-43ea-8a9d-d656802f1ade" (UID: "9b94b81c-0c49-43ea-8a9d-d656802f1ade"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:51:32.993731 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:32.993701    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b94b81c-0c49-43ea-8a9d-d656802f1ade-kube-api-access-8lncp" pod "9b94b81c-0c49-43ea-8a9d-d656802f1ade" (UID: "9b94b81c-0c49-43ea-8a9d-d656802f1ade"). InnerVolumeSpecName "kube-api-access-8lncp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:32.993860 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:32.993758    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-metrics-certs" pod "9b94b81c-0c49-43ea-8a9d-d656802f1ade" (UID: "9b94b81c-0c49-43ea-8a9d-d656802f1ade"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:51:32.993860 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:32.993783    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-maas-api-tls" pod "9b94b81c-0c49-43ea-8a9d-d656802f1ade" (UID: "9b94b81c-0c49-43ea-8a9d-d656802f1ade"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:51:33.092314 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:33.092276    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lncp\" (UniqueName: \"kubernetes.io/projected/9b94b81c-0c49-43ea-8a9d-d656802f1ade-kube-api-access-8lncp\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:33.092314 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:33.092310    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b94b81c-0c49-43ea-8a9d-d656802f1ade-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:33.092314 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:33.092324    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:33.092572 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:33.092336    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b94b81c-0c49-43ea-8a9d-d656802f1ade-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:33.378904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:33.378804    5415 generic.go:356] "Generic (PLEG): container finished" podID="9b94b81c-0c49-43ea-8a9d-d656802f1ade" containerID="ce104525b06f97e3ae255bb18d6812aec70a457217cbda44e72552cced99165c" exitCode=0
Oct 31 03:51:33.378904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:33.378861    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"
Oct 31 03:51:33.378904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:33.378866    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf" event={"ID":"9b94b81c-0c49-43ea-8a9d-d656802f1ade","Type":"ContainerDied","Data":"ce104525b06f97e3ae255bb18d6812aec70a457217cbda44e72552cced99165c"}
Oct 31 03:51:33.379169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:33.378942    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf" event={"ID":"9b94b81c-0c49-43ea-8a9d-d656802f1ade","Type":"ContainerDied","Data":"5c026042324e18cce86e50c15a643bb136904c8467b1afb1ca2953e563e5f627"}
Oct 31 03:51:33.379169 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:33.378964    5415 scope.go:122] "RemoveContainer" containerID="ce104525b06f97e3ae255bb18d6812aec70a457217cbda44e72552cced99165c"
Oct 31 03:51:33.389163 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:33.389144    5415 scope.go:122] "RemoveContainer" containerID="ce104525b06f97e3ae255bb18d6812aec70a457217cbda44e72552cced99165c"
Oct 31 03:51:33.389464 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:51:33.389424    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce104525b06f97e3ae255bb18d6812aec70a457217cbda44e72552cced99165c\": container with ID starting with ce104525b06f97e3ae255bb18d6812aec70a457217cbda44e72552cced99165c not found: ID does not exist" containerID="ce104525b06f97e3ae255bb18d6812aec70a457217cbda44e72552cced99165c"
Oct 31 03:51:33.389513 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:33.389475    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce104525b06f97e3ae255bb18d6812aec70a457217cbda44e72552cced99165c"} err="failed to get container status \"ce104525b06f97e3ae255bb18d6812aec70a457217cbda44e72552cced99165c\": rpc error: code = NotFound desc = could not find container \"ce104525b06f97e3ae255bb18d6812aec70a457217cbda44e72552cced99165c\": container with ID starting with ce104525b06f97e3ae255bb18d6812aec70a457217cbda44e72552cced99165c not found: ID does not exist"
Oct 31 03:51:33.535910 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:33.535856    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"]
Oct 31 03:51:33.539629 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:33.539599    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-14600f-84c9fcb5ff-lpftf"]
Oct 31 03:51:34.606706 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:34.606669    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b94b81c-0c49-43ea-8a9d-d656802f1ade" path="/var/lib/kubelet/pods/9b94b81c-0c49-43ea-8a9d-d656802f1ade/volumes"
Oct 31 03:51:37.378318 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:51:37.378291    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 03:51:46.176310 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:51:46.176202    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:52:05.909255 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:05.909216    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"]
Oct 31 03:52:05.909746 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:05.909419    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc" podUID="8fa81a6e-d0b4-43b6-8bba-87d56abba045" containerName="maas-api" containerID="cri-o://42b9112d1a81b956f6ba5c5c53d043f2b3194d3c869bc0fba0bd2cf98d46491e" gracePeriod=30
Oct 31 03:52:06.182315 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.178641    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:52:06.288572 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.288538    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa81a6e-d0b4-43b6-8bba-87d56abba045-kube-api-access-smcmm\" (UniqueName: \"kubernetes.io/projected/8fa81a6e-d0b4-43b6-8bba-87d56abba045-kube-api-access-smcmm\") pod \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") "
Oct 31 03:52:06.288789 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.288613    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa81a6e-d0b4-43b6-8bba-87d56abba045-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa81a6e-d0b4-43b6-8bba-87d56abba045-openshift-service-ca\") pod \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") "
Oct 31 03:52:06.288789 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.288667    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-metrics-certs\") pod \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") "
Oct 31 03:52:06.288789 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.288689    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-maas-api-tls\") pod \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\" (UID: \"8fa81a6e-d0b4-43b6-8bba-87d56abba045\") "
Oct 31 03:52:06.289361 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.289319    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa81a6e-d0b4-43b6-8bba-87d56abba045-openshift-service-ca" pod "8fa81a6e-d0b4-43b6-8bba-87d56abba045" (UID: "8fa81a6e-d0b4-43b6-8bba-87d56abba045"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:06.291054 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.291025    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-metrics-certs" pod "8fa81a6e-d0b4-43b6-8bba-87d56abba045" (UID: "8fa81a6e-d0b4-43b6-8bba-87d56abba045"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:52:06.291160 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.291071    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa81a6e-d0b4-43b6-8bba-87d56abba045-kube-api-access-smcmm" pod "8fa81a6e-d0b4-43b6-8bba-87d56abba045" (UID: "8fa81a6e-d0b4-43b6-8bba-87d56abba045"). InnerVolumeSpecName "kube-api-access-smcmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:06.291660 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.291402    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-maas-api-tls" pod "8fa81a6e-d0b4-43b6-8bba-87d56abba045" (UID: "8fa81a6e-d0b4-43b6-8bba-87d56abba045"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:52:06.390499 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.390464    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:06.390499 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.390499    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa81a6e-d0b4-43b6-8bba-87d56abba045-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:06.390721 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.390509    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smcmm\" (UniqueName: \"kubernetes.io/projected/8fa81a6e-d0b4-43b6-8bba-87d56abba045-kube-api-access-smcmm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:06.390721 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.390520    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa81a6e-d0b4-43b6-8bba-87d56abba045-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:06.582703 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.582674    5415 generic.go:356] "Generic (PLEG): container finished" podID="8fa81a6e-d0b4-43b6-8bba-87d56abba045" containerID="42b9112d1a81b956f6ba5c5c53d043f2b3194d3c869bc0fba0bd2cf98d46491e" exitCode=0
Oct 31 03:52:06.582864 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.582725    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"
Oct 31 03:52:06.582864 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.582756    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc" event={"ID":"8fa81a6e-d0b4-43b6-8bba-87d56abba045","Type":"ContainerDied","Data":"42b9112d1a81b956f6ba5c5c53d043f2b3194d3c869bc0fba0bd2cf98d46491e"}
Oct 31 03:52:06.582864 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.582810    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc" event={"ID":"8fa81a6e-d0b4-43b6-8bba-87d56abba045","Type":"ContainerDied","Data":"a0ccd6571c6fec77f53b3984f6cc120b30d476c7c1e68e0e9d38ea9df1a0f905"}
Oct 31 03:52:06.582864 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.582836    5415 scope.go:122] "RemoveContainer" containerID="42b9112d1a81b956f6ba5c5c53d043f2b3194d3c869bc0fba0bd2cf98d46491e"
Oct 31 03:52:06.593058 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.593025    5415 scope.go:122] "RemoveContainer" containerID="42b9112d1a81b956f6ba5c5c53d043f2b3194d3c869bc0fba0bd2cf98d46491e"
Oct 31 03:52:06.593367 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:52:06.593344    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b9112d1a81b956f6ba5c5c53d043f2b3194d3c869bc0fba0bd2cf98d46491e\": container with ID starting with 42b9112d1a81b956f6ba5c5c53d043f2b3194d3c869bc0fba0bd2cf98d46491e not found: ID does not exist" containerID="42b9112d1a81b956f6ba5c5c53d043f2b3194d3c869bc0fba0bd2cf98d46491e"
Oct 31 03:52:06.593419 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.593379    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b9112d1a81b956f6ba5c5c53d043f2b3194d3c869bc0fba0bd2cf98d46491e"} err="failed to get container status \"42b9112d1a81b956f6ba5c5c53d043f2b3194d3c869bc0fba0bd2cf98d46491e\": rpc error: code = NotFound desc = could not find container \"42b9112d1a81b956f6ba5c5c53d043f2b3194d3c869bc0fba0bd2cf98d46491e\": container with ID starting with 42b9112d1a81b956f6ba5c5c53d043f2b3194d3c869bc0fba0bd2cf98d46491e not found: ID does not exist"
Oct 31 03:52:06.737564 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.737526    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"]
Oct 31 03:52:06.741290 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:06.741257    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca15df-9fd78c59c-f4llc"]
Oct 31 03:52:08.607491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:08.607456    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa81a6e-d0b4-43b6-8bba-87d56abba045" path="/var/lib/kubelet/pods/8fa81a6e-d0b4-43b6-8bba-87d56abba045/volumes"
Oct 31 03:52:16.176298 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:52:16.176262    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:52:22.710911 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.710855    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"]
Oct 31 03:52:22.711403 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.711388    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8fa81a6e-d0b4-43b6-8bba-87d56abba045" containerName="maas-api"
Oct 31 03:52:22.711451 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.711405    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa81a6e-d0b4-43b6-8bba-87d56abba045" containerName="maas-api"
Oct 31 03:52:22.711451 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.711414    5415 cpu_manager.go:414] "containerMap: removing container" podUID="9b94b81c-0c49-43ea-8a9d-d656802f1ade" containerName="maas-api"
Oct 31 03:52:22.711451 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.711419    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b94b81c-0c49-43ea-8a9d-d656802f1ade" containerName="maas-api"
Oct 31 03:52:22.711550 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.711515    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa81a6e-d0b4-43b6-8bba-87d56abba045" containerName="maas-api"
Oct 31 03:52:22.711550 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.711524    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b94b81c-0c49-43ea-8a9d-d656802f1ade" containerName="maas-api"
Oct 31 03:52:22.716806 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.716783    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"
Oct 31 03:52:22.773435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.773384    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"]
Oct 31 03:52:22.786576 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.786540    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e251604-6dee-4850-be98-ad59356e3038-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6\" (UID: \"5e251604-6dee-4850-be98-ad59356e3038\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"
Oct 31 03:52:22.786818 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.786796    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbklw\" (UniqueName: \"kubernetes.io/projected/5e251604-6dee-4850-be98-ad59356e3038-kube-api-access-kbklw\") pod \"payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6\" (UID: \"5e251604-6dee-4850-be98-ad59356e3038\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"
Oct 31 03:52:22.827379 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.827341    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ab9b62\""
Oct 31 03:52:22.872330 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.872299    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ab9b62-dockercfg-9tjsf\""
Oct 31 03:52:22.887930 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.887860    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbklw\" (UniqueName: \"kubernetes.io/projected/5e251604-6dee-4850-be98-ad59356e3038-kube-api-access-kbklw\") pod \"payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6\" (UID: \"5e251604-6dee-4850-be98-ad59356e3038\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"
Oct 31 03:52:22.888106 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.887993    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e251604-6dee-4850-be98-ad59356e3038-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6\" (UID: \"5e251604-6dee-4850-be98-ad59356e3038\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"
Oct 31 03:52:22.888716 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.888698    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e251604-6dee-4850-be98-ad59356e3038-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6\" (UID: \"5e251604-6dee-4850-be98-ad59356e3038\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"
Oct 31 03:52:22.942063 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:22.942031    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbklw\" (UniqueName: \"kubernetes.io/projected/5e251604-6dee-4850-be98-ad59356e3038-kube-api-access-kbklw\") pod \"payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6\" (UID: \"5e251604-6dee-4850-be98-ad59356e3038\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"
Oct 31 03:52:23.035330 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:23.035204    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"
Oct 31 03:52:23.430123 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:23.430095    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"]
Oct 31 03:52:23.433269 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:52:23.433238    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e251604_6dee_4850_be98_ad59356e3038.slice/crio-3204169aace17159db20df69df3562a912be82750cbec415da537546a8f06056 WatchSource:0}: Error finding container 3204169aace17159db20df69df3562a912be82750cbec415da537546a8f06056: Status 404 returned error can't find the container with id 3204169aace17159db20df69df3562a912be82750cbec415da537546a8f06056
Oct 31 03:52:23.697835 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:23.697731    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6" event={"ID":"5e251604-6dee-4850-be98-ad59356e3038","Type":"ContainerStarted","Data":"cd29c96b1995bda709c1a2639d98126d5b4aa69ca8201c5af15b98640423db9a"}
Oct 31 03:52:23.697835 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:23.697794    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6" event={"ID":"5e251604-6dee-4850-be98-ad59356e3038","Type":"ContainerStarted","Data":"3204169aace17159db20df69df3562a912be82750cbec415da537546a8f06056"}
Oct 31 03:52:23.698119 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:23.697864    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"
Oct 31 03:52:23.806675 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:23.806602    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6" podStartSLOduration=1.806581859 podStartE2EDuration="1.806581859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:23.80603437 +0000 UTC m=+2685.366514104" watchObservedRunningTime="2026-10-31 03:52:23.806581859 +0000 UTC m=+2685.367061592"
Oct 31 03:52:34.707056 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:34.707019    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"
Oct 31 03:52:46.176166 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:52:46.176079    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:52:51.670157 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:51.670108    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"]
Oct 31 03:52:51.670619 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:51.670359    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" podUID="c2b20bf0-72db-4618-b347-c8db2b887b6d" containerName="istio-proxy" containerID="cri-o://b6686b29bbde7b4f95f2591fa5a903eccda7d3eac431e346c37f03ed0d9f2fdc" gracePeriod=30
Oct 31 03:52:56.884771 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:56.884737    5415 generic.go:356] "Generic (PLEG): container finished" podID="c2b20bf0-72db-4618-b347-c8db2b887b6d" containerID="b6686b29bbde7b4f95f2591fa5a903eccda7d3eac431e346c37f03ed0d9f2fdc" exitCode=0
Oct 31 03:52:56.885199 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:56.884805    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" event={"ID":"c2b20bf0-72db-4618-b347-c8db2b887b6d","Type":"ContainerDied","Data":"b6686b29bbde7b4f95f2591fa5a903eccda7d3eac431e346c37f03ed0d9f2fdc"}
Oct 31 03:52:56.946200 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:56.946131    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:52:57.041857 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.041815    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-socket\") pod \"c2b20bf0-72db-4618-b347-c8db2b887b6d\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") "
Oct 31 03:52:57.042097 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.041964    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-envoy\") pod \"c2b20bf0-72db-4618-b347-c8db2b887b6d\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") "
Oct 31 03:52:57.042097 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.041989    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-token\" (UniqueName: \"kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-token\") pod \"c2b20bf0-72db-4618-b347-c8db2b887b6d\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") "
Oct 31 03:52:57.042097 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042016    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b20bf0-72db-4618-b347-c8db2b887b6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b20bf0-72db-4618-b347-c8db2b887b6d-istiod-ca-cert\") pod \"c2b20bf0-72db-4618-b347-c8db2b887b6d\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") "
Oct 31 03:52:57.042097 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042038    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-credential-socket\") pod \"c2b20bf0-72db-4618-b347-c8db2b887b6d\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") "
Oct 31 03:52:57.042097 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042062    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-kube-api-access-prjsc\" (UniqueName: \"kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-kube-api-access-prjsc\") pod \"c2b20bf0-72db-4618-b347-c8db2b887b6d\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") "
Oct 31 03:52:57.042373 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042170    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-podinfo\") pod \"c2b20bf0-72db-4618-b347-c8db2b887b6d\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") "
Oct 31 03:52:57.042373 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042173    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-socket" pod "c2b20bf0-72db-4618-b347-c8db2b887b6d" (UID: "c2b20bf0-72db-4618-b347-c8db2b887b6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:57.042373 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042209    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-data\") pod \"c2b20bf0-72db-4618-b347-c8db2b887b6d\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") "
Oct 31 03:52:57.042373 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042240    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-certs\") pod \"c2b20bf0-72db-4618-b347-c8db2b887b6d\" (UID: \"c2b20bf0-72db-4618-b347-c8db2b887b6d\") "
Oct 31 03:52:57.042595 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042380    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-credential-socket" pod "c2b20bf0-72db-4618-b347-c8db2b887b6d" (UID: "c2b20bf0-72db-4618-b347-c8db2b887b6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:57.042595 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042513    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-data" pod "c2b20bf0-72db-4618-b347-c8db2b887b6d" (UID: "c2b20bf0-72db-4618-b347-c8db2b887b6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:57.042595 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042551    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b20bf0-72db-4618-b347-c8db2b887b6d-istiod-ca-cert" pod "c2b20bf0-72db-4618-b347-c8db2b887b6d" (UID: "c2b20bf0-72db-4618-b347-c8db2b887b6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:57.042767 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042729    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-certs" pod "c2b20bf0-72db-4618-b347-c8db2b887b6d" (UID: "c2b20bf0-72db-4618-b347-c8db2b887b6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:57.042900 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042865    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.042949 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042905    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.042949 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042921    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.042949 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042935    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b20bf0-72db-4618-b347-c8db2b887b6d-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.042949 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.042948    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.044340 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.044316    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-envoy" pod "c2b20bf0-72db-4618-b347-c8db2b887b6d" (UID: "c2b20bf0-72db-4618-b347-c8db2b887b6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:57.044477 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.044451    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-kube-api-access-prjsc" pod "c2b20bf0-72db-4618-b347-c8db2b887b6d" (UID: "c2b20bf0-72db-4618-b347-c8db2b887b6d"). InnerVolumeSpecName "kube-api-access-prjsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:57.044634 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.044614    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-token" pod "c2b20bf0-72db-4618-b347-c8db2b887b6d" (UID: "c2b20bf0-72db-4618-b347-c8db2b887b6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:57.044696 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.044647    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-podinfo" pod "c2b20bf0-72db-4618-b347-c8db2b887b6d" (UID: "c2b20bf0-72db-4618-b347-c8db2b887b6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:52:57.144227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.144194    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.144227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.144221    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.144227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.144230    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prjsc\" (UniqueName: \"kubernetes.io/projected/c2b20bf0-72db-4618-b347-c8db2b887b6d-kube-api-access-prjsc\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.144458 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.144241    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b20bf0-72db-4618-b347-c8db2b887b6d-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.892175 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.892142    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"
Oct 31 03:52:57.892668 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.892178    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw" event={"ID":"c2b20bf0-72db-4618-b347-c8db2b887b6d","Type":"ContainerDied","Data":"c9640d9642b729282dbd6e1158bfc865169c7c5b1101eaf1a9b20d30ebd02c43"}
Oct 31 03:52:57.892668 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:57.892253    5415 scope.go:122] "RemoveContainer" containerID="b6686b29bbde7b4f95f2591fa5a903eccda7d3eac431e346c37f03ed0d9f2fdc"
Oct 31 03:52:58.094631 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:58.094595    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"]
Oct 31 03:52:58.141834 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:58.141792    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hbjsw"]
Oct 31 03:52:58.610932 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:52:58.610863    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b20bf0-72db-4618-b347-c8db2b887b6d" path="/var/lib/kubelet/pods/c2b20bf0-72db-4618-b347-c8db2b887b6d/volumes"
Oct 31 03:53:08.654172 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.654135    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"]
Oct 31 03:53:08.654640 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.654628    5415 cpu_manager.go:414] "containerMap: removing container" podUID="c2b20bf0-72db-4618-b347-c8db2b887b6d" containerName="istio-proxy"
Oct 31 03:53:08.654703 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.654641    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b20bf0-72db-4618-b347-c8db2b887b6d" containerName="istio-proxy"
Oct 31 03:53:08.654770 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.654761    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b20bf0-72db-4618-b347-c8db2b887b6d" containerName="istio-proxy"
Oct 31 03:53:08.657859 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.657837    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"
Oct 31 03:53:08.707528 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.707494    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-a5f70c\""
Oct 31 03:53:08.754526 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.754491    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-a5f70c-dockercfg-wfs9f\""
Oct 31 03:53:08.756904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.756865    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"]
Oct 31 03:53:08.779508 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.779469    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e6897c-0910-4afe-bdbb-7a19250f6735-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9\" (UID: \"35e6897c-0910-4afe-bdbb-7a19250f6735\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"
Oct 31 03:53:08.779731 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.779706    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk2zz\" (UniqueName: \"kubernetes.io/projected/35e6897c-0910-4afe-bdbb-7a19250f6735-kube-api-access-lk2zz\") pod \"payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9\" (UID: \"35e6897c-0910-4afe-bdbb-7a19250f6735\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"
Oct 31 03:53:08.880835 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.880792    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk2zz\" (UniqueName: \"kubernetes.io/projected/35e6897c-0910-4afe-bdbb-7a19250f6735-kube-api-access-lk2zz\") pod \"payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9\" (UID: \"35e6897c-0910-4afe-bdbb-7a19250f6735\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"
Oct 31 03:53:08.881057 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.880927    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e6897c-0910-4afe-bdbb-7a19250f6735-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9\" (UID: \"35e6897c-0910-4afe-bdbb-7a19250f6735\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"
Oct 31 03:53:08.881654 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.881632    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e6897c-0910-4afe-bdbb-7a19250f6735-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9\" (UID: \"35e6897c-0910-4afe-bdbb-7a19250f6735\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"
Oct 31 03:53:08.936245 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.936172    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk2zz\" (UniqueName: \"kubernetes.io/projected/35e6897c-0910-4afe-bdbb-7a19250f6735-kube-api-access-lk2zz\") pod \"payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9\" (UID: \"35e6897c-0910-4afe-bdbb-7a19250f6735\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"
Oct 31 03:53:08.971482 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:08.971444    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"
Oct 31 03:53:09.339129 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:09.339099    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"]
Oct 31 03:53:09.341459 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:53:09.341431    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e6897c_0910_4afe_bdbb_7a19250f6735.slice/crio-fb63e7b298db79a16b5f6d6ddf942a6b3ec8f6a8c030a4edf1aec25556528a23 WatchSource:0}: Error finding container fb63e7b298db79a16b5f6d6ddf942a6b3ec8f6a8c030a4edf1aec25556528a23: Status 404 returned error can't find the container with id fb63e7b298db79a16b5f6d6ddf942a6b3ec8f6a8c030a4edf1aec25556528a23
Oct 31 03:53:09.962451 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:09.962413    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9" event={"ID":"35e6897c-0910-4afe-bdbb-7a19250f6735","Type":"ContainerStarted","Data":"71ab76403b72bc308cf01f73ec8b898be077df644ad35cd2e422485b0dfd2626"}
Oct 31 03:53:09.962969 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:09.962464    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9" event={"ID":"35e6897c-0910-4afe-bdbb-7a19250f6735","Type":"ContainerStarted","Data":"fb63e7b298db79a16b5f6d6ddf942a6b3ec8f6a8c030a4edf1aec25556528a23"}
Oct 31 03:53:09.962969 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:09.962501    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"
Oct 31 03:53:10.070257 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.070207    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9" podStartSLOduration=2.070195285 podStartE2EDuration="2.070195285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:10.070091816 +0000 UTC m=+2731.630571547" watchObservedRunningTime="2026-10-31 03:53:10.070195285 +0000 UTC m=+2731.630675016"
Oct 31 03:53:10.505957 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.505918    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"]
Oct 31 03:53:10.509904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.509870    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:10.559180 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.559142    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ab9b62-serving-cert\""
Oct 31 03:53:10.599026 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.598985    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad16f86-742e-4a4a-8e24-9de588505c03-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:10.599215 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.599041    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:10.599263 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.599231    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sgkw\" (UniqueName: \"kubernetes.io/projected/7ad16f86-742e-4a4a-8e24-9de588505c03-kube-api-access-7sgkw\") pod \"maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:10.599345 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.599329    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-metrics-certs\") pod \"maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:10.608368 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.608337    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"]
Oct 31 03:53:10.700479 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.700430    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad16f86-742e-4a4a-8e24-9de588505c03-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:10.700654 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.700492    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:10.700654 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.700579    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sgkw\" (UniqueName: \"kubernetes.io/projected/7ad16f86-742e-4a4a-8e24-9de588505c03-kube-api-access-7sgkw\") pod \"maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:10.700654 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:53:10.700637    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-serving-cert: secret "maas-api-e2e-praxis-a-ab9b62-serving-cert" not found
Oct 31 03:53:10.700863 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:53:10.700713    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-maas-api-tls podName:7ad16f86-742e-4a4a-8e24-9de588505c03 nodeName:}" failed. No retries permitted until 2026-10-31 03:53:11.200689231 +0000 UTC m=+2732.761168940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-maas-api-tls") pod "maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b" (UID: "7ad16f86-742e-4a4a-8e24-9de588505c03") : secret "maas-api-e2e-praxis-a-ab9b62-serving-cert" not found
Oct 31 03:53:10.700977 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.700640    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-metrics-certs\") pod \"maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:10.701269 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.701244    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad16f86-742e-4a4a-8e24-9de588505c03-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:10.703660 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.703635    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-metrics-certs\") pod \"maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:10.755642 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:10.755605    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sgkw\" (UniqueName: \"kubernetes.io/projected/7ad16f86-742e-4a4a-8e24-9de588505c03-kube-api-access-7sgkw\") pod \"maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:11.207802 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:11.207758    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:11.210325 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:11.210302    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:11.424822 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:11.424784    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:11.795331 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:11.795305    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"]
Oct 31 03:53:11.797850 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:53:11.797818    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad16f86_742e_4a4a_8e24_9de588505c03.slice/crio-b5b1b1950c7e4dcc3321339aed23c131a6d6a442d61043ce62314e8aa08a6126 WatchSource:0}: Error finding container b5b1b1950c7e4dcc3321339aed23c131a6d6a442d61043ce62314e8aa08a6126: Status 404 returned error can't find the container with id b5b1b1950c7e4dcc3321339aed23c131a6d6a442d61043ce62314e8aa08a6126
Oct 31 03:53:11.974572 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:11.974538    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b" event={"ID":"7ad16f86-742e-4a4a-8e24-9de588505c03","Type":"ContainerStarted","Data":"b5b1b1950c7e4dcc3321339aed23c131a6d6a442d61043ce62314e8aa08a6126"}
Oct 31 03:53:12.982860 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:12.982814    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b" event={"ID":"7ad16f86-742e-4a4a-8e24-9de588505c03","Type":"ContainerStarted","Data":"dfb267bc7bb7ea17cea5ea007facddb8b7f8d801ffed8580abc5fd5c8239456d"}
Oct 31 03:53:12.983276 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:12.983038    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:13.094940 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:13.094858    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b" podStartSLOduration=2.038282231 podStartE2EDuration="3.094833913s" totalImagesPullingTime="1.056551682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:13.093663449 +0000 UTC m=+2734.654143209" watchObservedRunningTime="2026-10-31 03:53:13.094833913 +0000 UTC m=+2734.655313643"
Oct 31 03:53:16.175929 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:53:16.175858    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:53:18.993782 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:18.993754    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 03:53:20.971143 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:20.971113    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"
Oct 31 03:53:36.039335 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.039290    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"]
Oct 31 03:53:36.043936 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.043911    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:36.092852 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.092819    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-502ec2-serving-cert\""
Oct 31 03:53:36.142276 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.142244    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"]
Oct 31 03:53:36.212464 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.212427    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvlck\" (UniqueName: \"kubernetes.io/projected/abc2f709-4ac0-40c9-bea6-5306df98321c-kube-api-access-mvlck\") pod \"maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:36.212658 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.212479    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc2f709-4ac0-40c9-bea6-5306df98321c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:36.212658 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.212607    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:36.212790 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.212766    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:36.314281 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.314187    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvlck\" (UniqueName: \"kubernetes.io/projected/abc2f709-4ac0-40c9-bea6-5306df98321c-kube-api-access-mvlck\") pod \"maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:36.314281 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.314236    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc2f709-4ac0-40c9-bea6-5306df98321c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:36.314281 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.314280    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:36.314522 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.314367    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:36.315018 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.314989    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc2f709-4ac0-40c9-bea6-5306df98321c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:36.316955 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.316929    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:36.317086 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.317014    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:36.371550 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.371514    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvlck\" (UniqueName: \"kubernetes.io/projected/abc2f709-4ac0-40c9-bea6-5306df98321c-kube-api-access-mvlck\") pod \"maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:36.657816 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:36.657772    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:37.025308 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:37.025281    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"]
Oct 31 03:53:37.027809 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:53:37.027775    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc2f709_4ac0_40c9_bea6_5306df98321c.slice/crio-e917c77be1c3ce519f77c10056b01ac2f03b0f3a22715e996ce9fd195ceb387a WatchSource:0}: Error finding container e917c77be1c3ce519f77c10056b01ac2f03b0f3a22715e996ce9fd195ceb387a: Status 404 returned error can't find the container with id e917c77be1c3ce519f77c10056b01ac2f03b0f3a22715e996ce9fd195ceb387a
Oct 31 03:53:37.117207 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:37.117170    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2" event={"ID":"abc2f709-4ac0-40c9-bea6-5306df98321c","Type":"ContainerStarted","Data":"e917c77be1c3ce519f77c10056b01ac2f03b0f3a22715e996ce9fd195ceb387a"}
Oct 31 03:53:38.124288 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:38.124244    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2" event={"ID":"abc2f709-4ac0-40c9-bea6-5306df98321c","Type":"ContainerStarted","Data":"f8cbe6888d8f3f675a6b2fd264b289bfffe1370ca3d902d27a08404c0134b2a0"}
Oct 31 03:53:38.124782 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:38.124315    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:38.237009 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:38.236897    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2" podStartSLOduration=2.304422679 podStartE2EDuration="3.236862665s" totalImagesPullingTime="932.439986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:38.236323917 +0000 UTC m=+2759.796803648" watchObservedRunningTime="2026-10-31 03:53:38.236862665 +0000 UTC m=+2759.797342412"
Oct 31 03:53:39.938940 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:39.938900    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"]
Oct 31 03:53:39.944620 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:39.944594    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"
Oct 31 03:53:40.041387 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:40.041332    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-dffe64\""
Oct 31 03:53:40.041587 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:40.041435    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-dffe64-dockercfg-hzl5p\""
Oct 31 03:53:40.050174 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:40.044683    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"]
Oct 31 03:53:40.060595 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:40.060555    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdtzp\" (UniqueName: \"kubernetes.io/projected/c0535859-46d6-489e-b7a4-8601db014cfb-kube-api-access-pdtzp\") pod \"payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp\" (UID: \"c0535859-46d6-489e-b7a4-8601db014cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"
Oct 31 03:53:40.060778 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:40.060622    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0535859-46d6-489e-b7a4-8601db014cfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp\" (UID: \"c0535859-46d6-489e-b7a4-8601db014cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"
Oct 31 03:53:40.161795 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:40.161756    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdtzp\" (UniqueName: \"kubernetes.io/projected/c0535859-46d6-489e-b7a4-8601db014cfb-kube-api-access-pdtzp\") pod \"payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp\" (UID: \"c0535859-46d6-489e-b7a4-8601db014cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"
Oct 31 03:53:40.162023 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:40.161817    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0535859-46d6-489e-b7a4-8601db014cfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp\" (UID: \"c0535859-46d6-489e-b7a4-8601db014cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"
Oct 31 03:53:40.162621 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:40.162600    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0535859-46d6-489e-b7a4-8601db014cfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp\" (UID: \"c0535859-46d6-489e-b7a4-8601db014cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"
Oct 31 03:53:40.219575 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:40.219505    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdtzp\" (UniqueName: \"kubernetes.io/projected/c0535859-46d6-489e-b7a4-8601db014cfb-kube-api-access-pdtzp\") pod \"payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp\" (UID: \"c0535859-46d6-489e-b7a4-8601db014cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"
Oct 31 03:53:40.259757 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:40.259718    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"
Oct 31 03:53:40.634721 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:40.634680    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"]
Oct 31 03:53:40.637250 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:53:40.637211    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0535859_46d6_489e_b7a4_8601db014cfb.slice/crio-ade285edc5a97d514aa6dfcd3eeee5bdbbc0de4ca0dd78149d8e8a2d3170c1af WatchSource:0}: Error finding container ade285edc5a97d514aa6dfcd3eeee5bdbbc0de4ca0dd78149d8e8a2d3170c1af: Status 404 returned error can't find the container with id ade285edc5a97d514aa6dfcd3eeee5bdbbc0de4ca0dd78149d8e8a2d3170c1af
Oct 31 03:53:41.141955 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:41.141917    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp" event={"ID":"c0535859-46d6-489e-b7a4-8601db014cfb","Type":"ContainerStarted","Data":"04463094e8dc6fb1644f8dc13dac509cc24b4aa0474f8729e40d5f4faa426a21"}
Oct 31 03:53:41.142494 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:41.141977    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp" event={"ID":"c0535859-46d6-489e-b7a4-8601db014cfb","Type":"ContainerStarted","Data":"ade285edc5a97d514aa6dfcd3eeee5bdbbc0de4ca0dd78149d8e8a2d3170c1af"}
Oct 31 03:53:41.142494 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:41.142000    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"
Oct 31 03:53:41.251829 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:41.251776    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp" podStartSLOduration=2.251760636 podStartE2EDuration="2.251760636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:41.249611508 +0000 UTC m=+2762.810091259" watchObservedRunningTime="2026-10-31 03:53:41.251760636 +0000 UTC m=+2762.812240367"
Oct 31 03:53:44.135848 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:44.135817    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 03:53:46.176376 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:53:46.176336    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:53:46.976615 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:46.976571    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"]
Oct 31 03:53:46.981262 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:46.981235    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.030888 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.030851    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-c0d50c-serving-cert\""
Oct 31 03:53:47.038071 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.038040    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"]
Oct 31 03:53:47.045476 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.045436    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e886c2bd-b363-480c-95c4-b889527cfa22-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.045646 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.045586    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9cc\" (UniqueName: \"kubernetes.io/projected/e886c2bd-b363-480c-95c4-b889527cfa22-kube-api-access-2z9cc\") pod \"maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.045719 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.045697    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.045813 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.045796    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.146984 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.146948    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.147162 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.147035    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.147162 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.147152    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e886c2bd-b363-480c-95c4-b889527cfa22-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.147253 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:53:47.147159    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-serving-cert: secret "maas-api-e2e-shared-b-w4-c0d50c-serving-cert" not found
Oct 31 03:53:47.147314 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:53:47.147252    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-maas-api-tls podName:e886c2bd-b363-480c-95c4-b889527cfa22 nodeName:}" failed. No retries permitted until 2026-10-31 03:53:47.647227767 +0000 UTC m=+2769.207707481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-maas-api-tls") pod "maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d" (UID: "e886c2bd-b363-480c-95c4-b889527cfa22") : secret "maas-api-e2e-shared-b-w4-c0d50c-serving-cert" not found
Oct 31 03:53:47.147314 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.147278    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9cc\" (UniqueName: \"kubernetes.io/projected/e886c2bd-b363-480c-95c4-b889527cfa22-kube-api-access-2z9cc\") pod \"maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.148012 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.147980    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e886c2bd-b363-480c-95c4-b889527cfa22-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.149926 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.149898    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.204574 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.204537    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9cc\" (UniqueName: \"kubernetes.io/projected/e886c2bd-b363-480c-95c4-b889527cfa22-kube-api-access-2z9cc\") pod \"maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.654469 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.654427    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.656951 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.656926    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:47.894640 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:47.894595    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:48.268227 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:48.268190    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"]
Oct 31 03:53:48.269524 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:53:48.269490    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode886c2bd_b363_480c_95c4_b889527cfa22.slice/crio-fd1e5d9dd555fca09c3009ab9bc7143dba38b81188773d07fe463321afa7e48b WatchSource:0}: Error finding container fd1e5d9dd555fca09c3009ab9bc7143dba38b81188773d07fe463321afa7e48b: Status 404 returned error can't find the container with id fd1e5d9dd555fca09c3009ab9bc7143dba38b81188773d07fe463321afa7e48b
Oct 31 03:53:49.196776 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:49.196729    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d" event={"ID":"e886c2bd-b363-480c-95c4-b889527cfa22","Type":"ContainerStarted","Data":"fd1e5d9dd555fca09c3009ab9bc7143dba38b81188773d07fe463321afa7e48b"}
Oct 31 03:53:50.203624 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:50.203590    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d" event={"ID":"e886c2bd-b363-480c-95c4-b889527cfa22","Type":"ContainerStarted","Data":"2e194cd144ddb7a31575cd5e434233efa0b2e33cbd13aa6af49fe9a2d3d9b486"}
Oct 31 03:53:50.204218 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:50.203674    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:53:50.320364 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:50.320305    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d" podStartSLOduration=3.445493413 podStartE2EDuration="4.320289829s" totalImagesPullingTime="874.796416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:50.318152085 +0000 UTC m=+2771.878631835" watchObservedRunningTime="2026-10-31 03:53:50.320289829 +0000 UTC m=+2771.880769560"
Oct 31 03:53:52.151327 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:52.151296    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"
Oct 31 03:53:56.215672 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:53:56.215642    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 03:54:16.176009 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:54:16.175928    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:54:36.893908 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:36.893845    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"]
Oct 31 03:54:36.898616 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:36.898581    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"
Oct 31 03:54:36.948255 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:36.948218    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-8b85a6\""
Oct 31 03:54:36.996694 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:36.995495    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-8b85a6-dockercfg-8cj2q\""
Oct 31 03:54:37.000102 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:36.999199    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"]
Oct 31 03:54:37.017238 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.017196    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"]
Oct 31 03:54:37.027221 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.027187    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"
Oct 31 03:54:37.052605 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.052563    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910e9857-1a61-41b9-a228-b39423e20d14-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294\" (UID: \"910e9857-1a61-41b9-a228-b39423e20d14\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"
Oct 31 03:54:37.052988 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.052957    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v82l2\" (UniqueName: \"kubernetes.io/projected/910e9857-1a61-41b9-a228-b39423e20d14-kube-api-access-v82l2\") pod \"payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294\" (UID: \"910e9857-1a61-41b9-a228-b39423e20d14\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"
Oct 31 03:54:37.053110 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.053008    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z\" (UID: \"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"
Oct 31 03:54:37.053110 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.053048    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ftmz\" (UniqueName: \"kubernetes.io/projected/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-kube-api-access-4ftmz\") pod \"payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z\" (UID: \"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"
Oct 31 03:54:37.077649 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.077611    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"]
Oct 31 03:54:37.154093 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.153990    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v82l2\" (UniqueName: \"kubernetes.io/projected/910e9857-1a61-41b9-a228-b39423e20d14-kube-api-access-v82l2\") pod \"payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294\" (UID: \"910e9857-1a61-41b9-a228-b39423e20d14\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"
Oct 31 03:54:37.154093 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.154054    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z\" (UID: \"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"
Oct 31 03:54:37.154093 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.154090    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ftmz\" (UniqueName: \"kubernetes.io/projected/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-kube-api-access-4ftmz\") pod \"payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z\" (UID: \"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"
Oct 31 03:54:37.154392 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.154194    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910e9857-1a61-41b9-a228-b39423e20d14-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294\" (UID: \"910e9857-1a61-41b9-a228-b39423e20d14\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"
Oct 31 03:54:37.154834 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.154811    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z\" (UID: \"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"
Oct 31 03:54:37.154892 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.154821    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910e9857-1a61-41b9-a228-b39423e20d14-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294\" (UID: \"910e9857-1a61-41b9-a228-b39423e20d14\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"
Oct 31 03:54:37.208420 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.208385    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ftmz\" (UniqueName: \"kubernetes.io/projected/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-kube-api-access-4ftmz\") pod \"payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z\" (UID: \"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"
Oct 31 03:54:37.208606 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.208458    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v82l2\" (UniqueName: \"kubernetes.io/projected/910e9857-1a61-41b9-a228-b39423e20d14-kube-api-access-v82l2\") pod \"payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294\" (UID: \"910e9857-1a61-41b9-a228-b39423e20d14\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"
Oct 31 03:54:37.213308 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.213277    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"
Oct 31 03:54:37.346856 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.346812    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"
Oct 31 03:54:37.583995 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.583965    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"]
Oct 31 03:54:37.586347 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:54:37.586318    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910e9857_1a61_41b9_a228_b39423e20d14.slice/crio-e3f3c14bf93df58b4647523e889e70fa3e11cf03c19d031de0e08f7a65154cd4 WatchSource:0}: Error finding container e3f3c14bf93df58b4647523e889e70fa3e11cf03c19d031de0e08f7a65154cd4: Status 404 returned error can't find the container with id e3f3c14bf93df58b4647523e889e70fa3e11cf03c19d031de0e08f7a65154cd4
Oct 31 03:54:37.720695 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:37.720668    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"]
Oct 31 03:54:37.722985 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:54:37.722948    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbac2bb_21dd_483b_9cbc_d25ddcb6282c.slice/crio-e1272a1278f0d18f5dffac81286e953c236b11b26d3fb5bab04957f16a34c640 WatchSource:0}: Error finding container e1272a1278f0d18f5dffac81286e953c236b11b26d3fb5bab04957f16a34c640: Status 404 returned error can't find the container with id e1272a1278f0d18f5dffac81286e953c236b11b26d3fb5bab04957f16a34c640
Oct 31 03:54:38.495427 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:38.495377    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z" event={"ID":"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c","Type":"ContainerStarted","Data":"7deefeb825c8d72574f77b6829d3975a5807cb947e5da39672921cbad286b6fe"}
Oct 31 03:54:38.495942 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:38.495442    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z" event={"ID":"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c","Type":"ContainerStarted","Data":"e1272a1278f0d18f5dffac81286e953c236b11b26d3fb5bab04957f16a34c640"}
Oct 31 03:54:38.495942 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:38.495467    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"
Oct 31 03:54:38.497110 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:38.497084    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294" event={"ID":"910e9857-1a61-41b9-a228-b39423e20d14","Type":"ContainerStarted","Data":"b30df98d7ec9a4f209d5fcd37b98a3d47c9436f365c17cb1e2be5437d1d41338"}
Oct 31 03:54:38.497233 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:38.497124    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294" event={"ID":"910e9857-1a61-41b9-a228-b39423e20d14","Type":"ContainerStarted","Data":"e3f3c14bf93df58b4647523e889e70fa3e11cf03c19d031de0e08f7a65154cd4"}
Oct 31 03:54:38.497233 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:38.497153    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"
Oct 31 03:54:38.616553 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:38.616493    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z" podStartSLOduration=2.616475325 podStartE2EDuration="2.616475325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:38.614336404 +0000 UTC m=+2820.174816152" watchObservedRunningTime="2026-10-31 03:54:38.616475325 +0000 UTC m=+2820.176955057"
Oct 31 03:54:38.727070 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:38.727018    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294" podStartSLOduration=2.727004183 podStartE2EDuration="2.727004183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:38.725948537 +0000 UTC m=+2820.286428268" watchObservedRunningTime="2026-10-31 03:54:38.727004183 +0000 UTC m=+2820.287483914"
Oct 31 03:54:44.607209 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.607177    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"]
Oct 31 03:54:44.612610 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.612585    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:44.642491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.642457    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-maas-api-tls\") pod \"maas-api-e2e-autores-dffe64-b57db78b5-t8rxs\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:44.642491 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.642497    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xctgw\" (UniqueName: \"kubernetes.io/projected/426d21ff-8092-4c91-85f1-76bdcd3e58b8-kube-api-access-xctgw\") pod \"maas-api-e2e-autores-dffe64-b57db78b5-t8rxs\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:44.642750 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.642520    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-metrics-certs\") pod \"maas-api-e2e-autores-dffe64-b57db78b5-t8rxs\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:44.642750 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.642673    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d21ff-8092-4c91-85f1-76bdcd3e58b8-openshift-service-ca\") pod \"maas-api-e2e-autores-dffe64-b57db78b5-t8rxs\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:44.666050 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.666019    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-dffe64-serving-cert\""
Oct 31 03:54:44.715609 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.715570    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"]
Oct 31 03:54:44.743707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.743664    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-maas-api-tls\") pod \"maas-api-e2e-autores-dffe64-b57db78b5-t8rxs\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:44.743707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.743705    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xctgw\" (UniqueName: \"kubernetes.io/projected/426d21ff-8092-4c91-85f1-76bdcd3e58b8-kube-api-access-xctgw\") pod \"maas-api-e2e-autores-dffe64-b57db78b5-t8rxs\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:44.743991 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.743727    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-metrics-certs\") pod \"maas-api-e2e-autores-dffe64-b57db78b5-t8rxs\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:44.743991 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.743771    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d21ff-8092-4c91-85f1-76bdcd3e58b8-openshift-service-ca\") pod \"maas-api-e2e-autores-dffe64-b57db78b5-t8rxs\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:44.743991 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:54:44.743814    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-serving-cert: secret "maas-api-e2e-autores-dffe64-serving-cert" not found
Oct 31 03:54:44.743991 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:54:44.743916    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-maas-api-tls podName:426d21ff-8092-4c91-85f1-76bdcd3e58b8 nodeName:}" failed. No retries permitted until 2026-10-31 03:54:45.243862228 +0000 UTC m=+2826.804341941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-maas-api-tls") pod "maas-api-e2e-autores-dffe64-b57db78b5-t8rxs" (UID: "426d21ff-8092-4c91-85f1-76bdcd3e58b8") : secret "maas-api-e2e-autores-dffe64-serving-cert" not found
Oct 31 03:54:44.744566 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.744543    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d21ff-8092-4c91-85f1-76bdcd3e58b8-openshift-service-ca\") pod \"maas-api-e2e-autores-dffe64-b57db78b5-t8rxs\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:44.746367 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.746344    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-metrics-certs\") pod \"maas-api-e2e-autores-dffe64-b57db78b5-t8rxs\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:44.799471 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:44.799430    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xctgw\" (UniqueName: \"kubernetes.io/projected/426d21ff-8092-4c91-85f1-76bdcd3e58b8-kube-api-access-xctgw\") pod \"maas-api-e2e-autores-dffe64-b57db78b5-t8rxs\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:45.250672 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:45.250638    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-maas-api-tls\") pod \"maas-api-e2e-autores-dffe64-b57db78b5-t8rxs\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:45.253300 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:45.253276    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-maas-api-tls\") pod \"maas-api-e2e-autores-dffe64-b57db78b5-t8rxs\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:45.527118 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:45.527032    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:45.899714 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:45.899538    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"]
Oct 31 03:54:45.901289 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:54:45.901255    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426d21ff_8092_4c91_85f1_76bdcd3e58b8.slice/crio-4a92baab2538cb096d2b6c41b61d86bc55f900dad9f41c26a9a2197368f9ed86 WatchSource:0}: Error finding container 4a92baab2538cb096d2b6c41b61d86bc55f900dad9f41c26a9a2197368f9ed86: Status 404 returned error can't find the container with id 4a92baab2538cb096d2b6c41b61d86bc55f900dad9f41c26a9a2197368f9ed86
Oct 31 03:54:46.176532 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:54:46.176436    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:54:46.556007 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:46.555912    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs" event={"ID":"426d21ff-8092-4c91-85f1-76bdcd3e58b8","Type":"ContainerStarted","Data":"4a92baab2538cb096d2b6c41b61d86bc55f900dad9f41c26a9a2197368f9ed86"}
Oct 31 03:54:47.563530 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:47.563488    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs" event={"ID":"426d21ff-8092-4c91-85f1-76bdcd3e58b8","Type":"ContainerStarted","Data":"2c9fae770599826ce447fdf4aec5aa96dd9db190eb2180c4815cf358b57c1ca0"}
Oct 31 03:54:47.564041 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:47.563557    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:54:47.675908 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:47.675819    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs" podStartSLOduration=2.73467292 podStartE2EDuration="3.675804081s" totalImagesPullingTime="941.131161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:47.67348848 +0000 UTC m=+2829.233968230" watchObservedRunningTime="2026-10-31 03:54:47.675804081 +0000 UTC m=+2829.236283812"
Oct 31 03:54:49.506282 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:49.506252    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"
Oct 31 03:54:49.506685 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:49.506316    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"
Oct 31 03:54:53.575614 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:54:53.575583    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:55:04.606431 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.606401    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"]
Oct 31 03:55:04.610518 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.610496    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.660273 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.660238    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"]
Oct 31 03:55:04.691402 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.691364    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-model-cache\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.691402 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.691407    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frs4r\" (UniqueName: \"kubernetes.io/projected/78d5705e-0fee-477d-a5c1-c02375c63ea5-kube-api-access-frs4r\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.691674 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.691441    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-dshm\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.691674 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.691478    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-tmp-dir\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.691797 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.691692    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-home\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.691849 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.691801    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78d5705e-0fee-477d-a5c1-c02375c63ea5-tls-certs\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.704869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.704826    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2339ed\"/\"openshift-service-ca.crt\""
Oct 31 03:55:04.704869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.704839    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2339ed\"/\"kube-root-ca.crt\""
Oct 31 03:55:04.704869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.704905    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2339ed\"/\"default-dockercfg-cxp4c\""
Oct 31 03:55:04.706886 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.706861    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2339ed\"/\"auth-test-model-2339ed-kserve-self-signed-certs\""
Oct 31 03:55:04.792767 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.792728    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-home\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.792947 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.792814    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78d5705e-0fee-477d-a5c1-c02375c63ea5-tls-certs\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.793014 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.792982    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-model-cache\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.793060 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.793011    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frs4r\" (UniqueName: \"kubernetes.io/projected/78d5705e-0fee-477d-a5c1-c02375c63ea5-kube-api-access-frs4r\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.793099 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.793065    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-dshm\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.793152 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.793112    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-tmp-dir\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.793216 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.793193    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-home\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.793345 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.793326    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-model-cache\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.793500 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.793467    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-tmp-dir\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.795707 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.795673    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-dshm\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.796035 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.796004    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78d5705e-0fee-477d-a5c1-c02375c63ea5-tls-certs\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.847826 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.847784    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frs4r\" (UniqueName: \"kubernetes.io/projected/78d5705e-0fee-477d-a5c1-c02375c63ea5-kube-api-access-frs4r\") pod \"auth-test-model-2339ed-kserve-64788fb64f-trqls\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:04.923049 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:04.922960    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:05.297904 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:05.297867    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"]
Oct 31 03:55:05.299919 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:55:05.299865    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78d5705e_0fee_477d_a5c1_c02375c63ea5.slice/crio-f0486543cb3df68ee82fbf53448df21f9acb3b65e63779aa302a9d6c708a31c2 WatchSource:0}: Error finding container f0486543cb3df68ee82fbf53448df21f9acb3b65e63779aa302a9d6c708a31c2: Status 404 returned error can't find the container with id f0486543cb3df68ee82fbf53448df21f9acb3b65e63779aa302a9d6c708a31c2
Oct 31 03:55:05.677519 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:05.677481    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls" event={"ID":"78d5705e-0fee-477d-a5c1-c02375c63ea5","Type":"ContainerStarted","Data":"bf7d474ccf79aba5623120e852993e96e6b06f48f5946f3fb9db5a8db542504e"}
Oct 31 03:55:05.677962 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:05.677532    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls" event={"ID":"78d5705e-0fee-477d-a5c1-c02375c63ea5","Type":"ContainerStarted","Data":"f0486543cb3df68ee82fbf53448df21f9acb3b65e63779aa302a9d6c708a31c2"}
Oct 31 03:55:05.786972 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:05.786911    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls" podStartSLOduration=1.7868953319999998 podStartE2EDuration="1.786895332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:05.784236021 +0000 UTC m=+2847.344715751" watchObservedRunningTime="2026-10-31 03:55:05.786895332 +0000 UTC m=+2847.347375059"
Oct 31 03:55:05.923758 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:05.923696    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:14.923151 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:14.923113    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:14.936318 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:14.936288    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:15.750592 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:55:15.750560    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 03:55:16.176796 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:55:16.176754    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:55:46.176000 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:55:46.175916    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:56:11.114271 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.114223    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"]
Oct 31 03:56:11.118805 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.118781    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.168747 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.168709    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-861dfd\"/\"kube-root-ca.crt\""
Oct 31 03:56:11.169082 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.169058    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-861dfd\"/\"openshift-service-ca.crt\""
Oct 31 03:56:11.215753 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.215718    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-861dfd\"/\"default-dockercfg-vhm7j\""
Oct 31 03:56:11.215753 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.215718    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-861dfd\"/\"auth-test-model-861dfd-kserve-self-signed-certs\""
Oct 31 03:56:11.218645 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.218623    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"]
Oct 31 03:56:11.244510 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.244477    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-home\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.244688 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.244532    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tmp-dir\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.244688 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.244642    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-dshm\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.244792 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.244774    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tls-certs\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.244834 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.244819    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzsxn\" (UniqueName: \"kubernetes.io/projected/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-kube-api-access-wzsxn\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.244898 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.244847    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-model-cache\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.346630 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.346579    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tls-certs\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.346837 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.346655    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzsxn\" (UniqueName: \"kubernetes.io/projected/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-kube-api-access-wzsxn\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.346837 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.346686    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-model-cache\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.346837 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.346781    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-home\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.347041 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.346842    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tmp-dir\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.347041 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.346903    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-dshm\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.347348 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.347314    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-model-cache\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.347699 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.347585    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-home\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.347889 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.347851    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tmp-dir\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.353113 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.353086    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-dshm\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.353543 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.353514    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tls-certs\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.401095 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.401010    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzsxn\" (UniqueName: \"kubernetes.io/projected/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-kube-api-access-wzsxn\") pod \"auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") " pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.436409 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.436376    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:11.814924 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:11.814899    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"]
Oct 31 03:56:11.817188 ip-10-0-139-164 kubenswrapper[5415]: W1031 03:56:11.817156    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5f16c8f_af5e_47a0_b925_cf44e3e4fef4.slice/crio-7243c72a9211daf8a860e148135f0f660589477003f294252fbfebb6164dda20 WatchSource:0}: Error finding container 7243c72a9211daf8a860e148135f0f660589477003f294252fbfebb6164dda20: Status 404 returned error can't find the container with id 7243c72a9211daf8a860e148135f0f660589477003f294252fbfebb6164dda20
Oct 31 03:56:12.099699 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:12.099660    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww" event={"ID":"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4","Type":"ContainerStarted","Data":"d17171f2a7c8458ffbc77b02371b32b58a501912ec0b96c5c68b7df2ec3ea1d3"}
Oct 31 03:56:12.099699 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:12.099705    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww" event={"ID":"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4","Type":"ContainerStarted","Data":"7243c72a9211daf8a860e148135f0f660589477003f294252fbfebb6164dda20"}
Oct 31 03:56:12.210109 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:12.210055    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww" podStartSLOduration=1.210038046 podStartE2EDuration="1.210038046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:12.207517505 +0000 UTC m=+2913.767997236" watchObservedRunningTime="2026-10-31 03:56:12.210038046 +0000 UTC m=+2913.770517778"
Oct 31 03:56:12.436907 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:12.436787    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:16.178367 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:56:16.178328    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:56:21.436690 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:21.436650    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:21.450498 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:21.450470    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:22.171129 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:56:22.171098    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 03:56:46.176071 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:56:46.176027    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:57:16.175930 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:57:16.175821    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:57:46.176450 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:57:46.176413    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:58:03.062698 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:03.062648    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"]
Oct 31 03:58:03.063216 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:03.062944    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" podUID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" containerName="istio-proxy" containerID="cri-o://c9ef1ca18632d38edb7da8649c0275aa498401f2e8cedf56c258c054c3b833cc" gracePeriod=30
Oct 31 03:58:08.105365 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.105324    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:58:08.105766 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.105401    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" podUID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 03:58:08.327825 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.327751    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:58:08.417826 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.417791    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-podinfo\") pod \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") "
Oct 31 03:58:08.418008 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.417922    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-data\") pod \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") "
Oct 31 03:58:08.418008 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.417958    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-kube-api-access-dww24\" (UniqueName: \"kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-kube-api-access-dww24\") pod \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") "
Oct 31 03:58:08.418008 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.417999    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-credential-socket\") pod \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") "
Oct 31 03:58:08.418209 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418028    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-socket\") pod \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") "
Oct 31 03:58:08.418209 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418049    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-envoy\") pod \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") "
Oct 31 03:58:08.418209 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418067    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-token\" (UniqueName: \"kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-token\") pod \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") "
Oct 31 03:58:08.418209 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418129    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istiod-ca-cert\") pod \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") "
Oct 31 03:58:08.418209 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418204    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-certs\") pod \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\" (UID: \"f707fded-4ede-4eb3-99a2-9c16ccb99f50\") "
Oct 31 03:58:08.418464 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418338    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-data" pod "f707fded-4ede-4eb3-99a2-9c16ccb99f50" (UID: "f707fded-4ede-4eb3-99a2-9c16ccb99f50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:08.418464 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418379    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-socket" pod "f707fded-4ede-4eb3-99a2-9c16ccb99f50" (UID: "f707fded-4ede-4eb3-99a2-9c16ccb99f50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:08.418464 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418394    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-credential-socket" pod "f707fded-4ede-4eb3-99a2-9c16ccb99f50" (UID: "f707fded-4ede-4eb3-99a2-9c16ccb99f50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:08.418627 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418583    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-certs" pod "f707fded-4ede-4eb3-99a2-9c16ccb99f50" (UID: "f707fded-4ede-4eb3-99a2-9c16ccb99f50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:08.418712 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418673    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istiod-ca-cert" pod "f707fded-4ede-4eb3-99a2-9c16ccb99f50" (UID: "f707fded-4ede-4eb3-99a2-9c16ccb99f50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:58:08.418873 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418856    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.418977 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418895    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.418977 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418910    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.418977 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418923    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.418977 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.418937    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.420612 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.420585    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-envoy" pod "f707fded-4ede-4eb3-99a2-9c16ccb99f50" (UID: "f707fded-4ede-4eb3-99a2-9c16ccb99f50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:08.420728 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.420599    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-podinfo" pod "f707fded-4ede-4eb3-99a2-9c16ccb99f50" (UID: "f707fded-4ede-4eb3-99a2-9c16ccb99f50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:58:08.420790 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.420748    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-kube-api-access-dww24" pod "f707fded-4ede-4eb3-99a2-9c16ccb99f50" (UID: "f707fded-4ede-4eb3-99a2-9c16ccb99f50"). InnerVolumeSpecName "kube-api-access-dww24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:08.420790 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.420771    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-token" pod "f707fded-4ede-4eb3-99a2-9c16ccb99f50" (UID: "f707fded-4ede-4eb3-99a2-9c16ccb99f50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:08.520232 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.520193    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.520232 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.520230    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.520494 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.520245    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f707fded-4ede-4eb3-99a2-9c16ccb99f50-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.520494 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.520259    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dww24\" (UniqueName: \"kubernetes.io/projected/f707fded-4ede-4eb3-99a2-9c16ccb99f50-kube-api-access-dww24\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.829040 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.829005    5415 generic.go:356] "Generic (PLEG): container finished" podID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" containerID="c9ef1ca18632d38edb7da8649c0275aa498401f2e8cedf56c258c054c3b833cc" exitCode=0
Oct 31 03:58:08.829235 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.829059    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"
Oct 31 03:58:08.829235 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.829088    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" event={"ID":"f707fded-4ede-4eb3-99a2-9c16ccb99f50","Type":"ContainerDied","Data":"c9ef1ca18632d38edb7da8649c0275aa498401f2e8cedf56c258c054c3b833cc"}
Oct 31 03:58:08.829235 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.829153    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf" event={"ID":"f707fded-4ede-4eb3-99a2-9c16ccb99f50","Type":"ContainerDied","Data":"48495d534307354706408f5b3c6debea121560354fbb232665fc7f95666d0bc0"}
Oct 31 03:58:08.829235 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.829182    5415 scope.go:122] "RemoveContainer" containerID="c9ef1ca18632d38edb7da8649c0275aa498401f2e8cedf56c258c054c3b833cc"
Oct 31 03:58:08.842944 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.842922    5415 scope.go:122] "RemoveContainer" containerID="c9ef1ca18632d38edb7da8649c0275aa498401f2e8cedf56c258c054c3b833cc"
Oct 31 03:58:08.843262 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:58:08.843242    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ef1ca18632d38edb7da8649c0275aa498401f2e8cedf56c258c054c3b833cc\": container with ID starting with c9ef1ca18632d38edb7da8649c0275aa498401f2e8cedf56c258c054c3b833cc not found: ID does not exist" containerID="c9ef1ca18632d38edb7da8649c0275aa498401f2e8cedf56c258c054c3b833cc"
Oct 31 03:58:08.843313 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.843272    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ef1ca18632d38edb7da8649c0275aa498401f2e8cedf56c258c054c3b833cc"} err="failed to get container status \"c9ef1ca18632d38edb7da8649c0275aa498401f2e8cedf56c258c054c3b833cc\": rpc error: code = NotFound desc = could not find container \"c9ef1ca18632d38edb7da8649c0275aa498401f2e8cedf56c258c054c3b833cc\": container with ID starting with c9ef1ca18632d38edb7da8649c0275aa498401f2e8cedf56c258c054c3b833cc not found: ID does not exist"
Oct 31 03:58:08.987362 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:08.987264    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"]
Oct 31 03:58:09.033916 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:09.033846    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-lx6qf"]
Oct 31 03:58:10.605891 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:10.605852    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" path="/var/lib/kubelet/pods/f707fded-4ede-4eb3-99a2-9c16ccb99f50/volumes"
Oct 31 03:58:14.832079 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:14.832039    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"]
Oct 31 03:58:14.832554 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:14.832257    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf" podUID="c64a42a6-ace4-42e2-bf65-f834e9c4c263" containerName="istio-proxy" containerID="cri-o://432d02ac44ce018baae7501663149cd9990bbb6c0c3e96a75015fee4b4f0f38e" gracePeriod=30
Oct 31 03:58:16.180176 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:58:16.180139    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:58:19.731837 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:19.731798    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:58:19.732342 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:19.731863    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf" podUID="c64a42a6-ace4-42e2-bf65-f834e9c4c263" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:58:20.094313 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.094285    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:58:20.268631 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.268538    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-kube-api-access-jnhdb\" (UniqueName: \"kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-kube-api-access-jnhdb\") pod \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") "
Oct 31 03:58:20.268631 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.268590    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-token\" (UniqueName: \"kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-token\") pod \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") "
Oct 31 03:58:20.268869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.268740    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-credential-socket\") pod \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") "
Oct 31 03:58:20.268869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.268817    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-podinfo\") pod \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") "
Oct 31 03:58:20.268869 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.268859    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-certs\") pod \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") "
Oct 31 03:58:20.269078 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.268922    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-socket\") pod \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") "
Oct 31 03:58:20.269078 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.269032    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istiod-ca-cert\") pod \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") "
Oct 31 03:58:20.269078 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.269029    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-credential-socket" pod "c64a42a6-ace4-42e2-bf65-f834e9c4c263" (UID: "c64a42a6-ace4-42e2-bf65-f834e9c4c263"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:20.269078 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.269068    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-data\") pod \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") "
Oct 31 03:58:20.269284 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.269104    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-envoy\") pod \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\" (UID: \"c64a42a6-ace4-42e2-bf65-f834e9c4c263\") "
Oct 31 03:58:20.269284 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.269240    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-certs" pod "c64a42a6-ace4-42e2-bf65-f834e9c4c263" (UID: "c64a42a6-ace4-42e2-bf65-f834e9c4c263"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:20.270221 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.269694    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.270221 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.269721    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.270221 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.269792    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istiod-ca-cert" pod "c64a42a6-ace4-42e2-bf65-f834e9c4c263" (UID: "c64a42a6-ace4-42e2-bf65-f834e9c4c263"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:58:20.270221 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.270103    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-socket" pod "c64a42a6-ace4-42e2-bf65-f834e9c4c263" (UID: "c64a42a6-ace4-42e2-bf65-f834e9c4c263"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:20.270639 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.270277    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-data" pod "c64a42a6-ace4-42e2-bf65-f834e9c4c263" (UID: "c64a42a6-ace4-42e2-bf65-f834e9c4c263"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:20.271396 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.271368    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-kube-api-access-jnhdb" pod "c64a42a6-ace4-42e2-bf65-f834e9c4c263" (UID: "c64a42a6-ace4-42e2-bf65-f834e9c4c263"). InnerVolumeSpecName "kube-api-access-jnhdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:20.271574 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.271554    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-token" pod "c64a42a6-ace4-42e2-bf65-f834e9c4c263" (UID: "c64a42a6-ace4-42e2-bf65-f834e9c4c263"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:20.272190 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.272159    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-envoy" pod "c64a42a6-ace4-42e2-bf65-f834e9c4c263" (UID: "c64a42a6-ace4-42e2-bf65-f834e9c4c263"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:20.272668 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.272632    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-podinfo" pod "c64a42a6-ace4-42e2-bf65-f834e9c4c263" (UID: "c64a42a6-ace4-42e2-bf65-f834e9c4c263"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:58:20.370860 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.370822    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.370860 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.370855    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.370860 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.370865    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.371142 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.370910    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.371142 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.370921    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.371142 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.370930    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnhdb\" (UniqueName: \"kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-kube-api-access-jnhdb\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.371142 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.370939    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64a42a6-ace4-42e2-bf65-f834e9c4c263-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.904833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.904739    5415 generic.go:356] "Generic (PLEG): container finished" podID="c64a42a6-ace4-42e2-bf65-f834e9c4c263" containerID="432d02ac44ce018baae7501663149cd9990bbb6c0c3e96a75015fee4b4f0f38e" exitCode=0
Oct 31 03:58:20.904833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.904803    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"
Oct 31 03:58:20.904833 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.904823    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf" event={"ID":"c64a42a6-ace4-42e2-bf65-f834e9c4c263","Type":"ContainerDied","Data":"432d02ac44ce018baae7501663149cd9990bbb6c0c3e96a75015fee4b4f0f38e"}
Oct 31 03:58:20.905607 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.904913    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf" event={"ID":"c64a42a6-ace4-42e2-bf65-f834e9c4c263","Type":"ContainerDied","Data":"11f0bb2d608e81825bf9f09fb988dc6c4169998778fc2f96244fe3124ddbdb9e"}
Oct 31 03:58:20.905607 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.904943    5415 scope.go:122] "RemoveContainer" containerID="432d02ac44ce018baae7501663149cd9990bbb6c0c3e96a75015fee4b4f0f38e"
Oct 31 03:58:20.915370 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.915347    5415 scope.go:122] "RemoveContainer" containerID="432d02ac44ce018baae7501663149cd9990bbb6c0c3e96a75015fee4b4f0f38e"
Oct 31 03:58:20.915706 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:58:20.915685    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432d02ac44ce018baae7501663149cd9990bbb6c0c3e96a75015fee4b4f0f38e\": container with ID starting with 432d02ac44ce018baae7501663149cd9990bbb6c0c3e96a75015fee4b4f0f38e not found: ID does not exist" containerID="432d02ac44ce018baae7501663149cd9990bbb6c0c3e96a75015fee4b4f0f38e"
Oct 31 03:58:20.915786 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:20.915718    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432d02ac44ce018baae7501663149cd9990bbb6c0c3e96a75015fee4b4f0f38e"} err="failed to get container status \"432d02ac44ce018baae7501663149cd9990bbb6c0c3e96a75015fee4b4f0f38e\": rpc error: code = NotFound desc = could not find container \"432d02ac44ce018baae7501663149cd9990bbb6c0c3e96a75015fee4b4f0f38e\": container with ID starting with 432d02ac44ce018baae7501663149cd9990bbb6c0c3e96a75015fee4b4f0f38e not found: ID does not exist"
Oct 31 03:58:21.064019 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:21.063971    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"]
Oct 31 03:58:21.066370 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:21.066343    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-w8fbf"]
Oct 31 03:58:22.606516 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:58:22.606481    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64a42a6-ace4-42e2-bf65-f834e9c4c263" path="/var/lib/kubelet/pods/c64a42a6-ace4-42e2-bf65-f834e9c4c263/volumes"
Oct 31 03:58:46.179506 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:58:46.179422    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:59:15.270084 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:15.270044    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"]
Oct 31 03:59:15.270613 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:15.270290    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h" podUID="19da59fb-854b-4e33-b635-c68551bf4179" containerName="istio-proxy" containerID="cri-o://4ce133667629a5036058142c7a207e5627c0e59e103a6fcb46aa5a35c59d40a7" gracePeriod=30
Oct 31 03:59:16.176771 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:59:16.176732    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:59:19.292185 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:19.292143    5415 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:59:19.292623 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:19.292210    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h" podUID="19da59fb-854b-4e33-b635-c68551bf4179" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:59:20.548903 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.548824    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:59:20.581962 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.581929    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-kube-api-access-7b5nj\" (UniqueName: \"kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-kube-api-access-7b5nj\") pod \"19da59fb-854b-4e33-b635-c68551bf4179\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") "
Oct 31 03:59:20.581962 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.581975    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-envoy\") pod \"19da59fb-854b-4e33-b635-c68551bf4179\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") "
Oct 31 03:59:20.582183 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.582003    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19da59fb-854b-4e33-b635-c68551bf4179-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19da59fb-854b-4e33-b635-c68551bf4179-istiod-ca-cert\") pod \"19da59fb-854b-4e33-b635-c68551bf4179\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") "
Oct 31 03:59:20.582183 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.582069    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-data\") pod \"19da59fb-854b-4e33-b635-c68551bf4179\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") "
Oct 31 03:59:20.582183 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.582161    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-socket\") pod \"19da59fb-854b-4e33-b635-c68551bf4179\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") "
Oct 31 03:59:20.582323 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.582245    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19da59fb-854b-4e33-b635-c68551bf4179-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19da59fb-854b-4e33-b635-c68551bf4179-istio-podinfo\") pod \"19da59fb-854b-4e33-b635-c68551bf4179\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") "
Oct 31 03:59:20.582323 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.582304    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-istio-token\" (UniqueName: \"kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-istio-token\") pod \"19da59fb-854b-4e33-b635-c68551bf4179\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") "
Oct 31 03:59:20.582435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.582347    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-credential-socket\") pod \"19da59fb-854b-4e33-b635-c68551bf4179\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") "
Oct 31 03:59:20.582435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.582351    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-data" pod "19da59fb-854b-4e33-b635-c68551bf4179" (UID: "19da59fb-854b-4e33-b635-c68551bf4179"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:20.582435 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.582398    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-certs\") pod \"19da59fb-854b-4e33-b635-c68551bf4179\" (UID: \"19da59fb-854b-4e33-b635-c68551bf4179\") "
Oct 31 03:59:20.582599 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.582474    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-socket" pod "19da59fb-854b-4e33-b635-c68551bf4179" (UID: "19da59fb-854b-4e33-b635-c68551bf4179"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:20.582599 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.582485    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19da59fb-854b-4e33-b635-c68551bf4179-istiod-ca-cert" pod "19da59fb-854b-4e33-b635-c68551bf4179" (UID: "19da59fb-854b-4e33-b635-c68551bf4179"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:20.582734 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.582711    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-credential-socket" pod "19da59fb-854b-4e33-b635-c68551bf4179" (UID: "19da59fb-854b-4e33-b635-c68551bf4179"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:20.582873 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.582847    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-certs" pod "19da59fb-854b-4e33-b635-c68551bf4179" (UID: "19da59fb-854b-4e33-b635-c68551bf4179"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:20.583004 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.582982    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:20.583080 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.583011    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:20.583080 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.583026    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:20.583080 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.583040    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19da59fb-854b-4e33-b635-c68551bf4179-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:20.583080 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.583054    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:20.584811 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.584787    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-envoy" pod "19da59fb-854b-4e33-b635-c68551bf4179" (UID: "19da59fb-854b-4e33-b635-c68551bf4179"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:20.584946 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.584813    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-kube-api-access-7b5nj" pod "19da59fb-854b-4e33-b635-c68551bf4179" (UID: "19da59fb-854b-4e33-b635-c68551bf4179"). InnerVolumeSpecName "kube-api-access-7b5nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:20.584946 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.584926    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-istio-token" pod "19da59fb-854b-4e33-b635-c68551bf4179" (UID: "19da59fb-854b-4e33-b635-c68551bf4179"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:20.585026 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.584958    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19da59fb-854b-4e33-b635-c68551bf4179-istio-podinfo" pod "19da59fb-854b-4e33-b635-c68551bf4179" (UID: "19da59fb-854b-4e33-b635-c68551bf4179"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:59:20.685060 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.684966    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19da59fb-854b-4e33-b635-c68551bf4179-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:20.685060 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.685004    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:20.685060 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.685047    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b5nj\" (UniqueName: \"kubernetes.io/projected/19da59fb-854b-4e33-b635-c68551bf4179-kube-api-access-7b5nj\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:20.685060 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:20.685071    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19da59fb-854b-4e33-b635-c68551bf4179-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.277224 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:21.277192    5415 generic.go:356] "Generic (PLEG): container finished" podID="19da59fb-854b-4e33-b635-c68551bf4179" containerID="4ce133667629a5036058142c7a207e5627c0e59e103a6fcb46aa5a35c59d40a7" exitCode=0
Oct 31 03:59:21.277426 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:21.277242    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"
Oct 31 03:59:21.277426 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:21.277274    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h" event={"ID":"19da59fb-854b-4e33-b635-c68551bf4179","Type":"ContainerDied","Data":"4ce133667629a5036058142c7a207e5627c0e59e103a6fcb46aa5a35c59d40a7"}
Oct 31 03:59:21.277426 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:21.277331    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h" event={"ID":"19da59fb-854b-4e33-b635-c68551bf4179","Type":"ContainerDied","Data":"e12c8f5ecf153cf5a1bfb2c3c75c47a613c59a7ea1dd20f8cf267b5a14eac173"}
Oct 31 03:59:21.277426 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:21.277355    5415 scope.go:122] "RemoveContainer" containerID="4ce133667629a5036058142c7a207e5627c0e59e103a6fcb46aa5a35c59d40a7"
Oct 31 03:59:21.289215 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:21.289194    5415 scope.go:122] "RemoveContainer" containerID="4ce133667629a5036058142c7a207e5627c0e59e103a6fcb46aa5a35c59d40a7"
Oct 31 03:59:21.289588 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:59:21.289554    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce133667629a5036058142c7a207e5627c0e59e103a6fcb46aa5a35c59d40a7\": container with ID starting with 4ce133667629a5036058142c7a207e5627c0e59e103a6fcb46aa5a35c59d40a7 not found: ID does not exist" containerID="4ce133667629a5036058142c7a207e5627c0e59e103a6fcb46aa5a35c59d40a7"
Oct 31 03:59:21.289703 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:21.289598    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce133667629a5036058142c7a207e5627c0e59e103a6fcb46aa5a35c59d40a7"} err="failed to get container status \"4ce133667629a5036058142c7a207e5627c0e59e103a6fcb46aa5a35c59d40a7\": rpc error: code = NotFound desc = could not find container \"4ce133667629a5036058142c7a207e5627c0e59e103a6fcb46aa5a35c59d40a7\": container with ID starting with 4ce133667629a5036058142c7a207e5627c0e59e103a6fcb46aa5a35c59d40a7 not found: ID does not exist"
Oct 31 03:59:21.433480 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:21.433426    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"]
Oct 31 03:59:21.440071 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:21.440019    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-2fn6h"]
Oct 31 03:59:22.606043 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:22.605999    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19da59fb-854b-4e33-b635-c68551bf4179" path="/var/lib/kubelet/pods/19da59fb-854b-4e33-b635-c68551bf4179/volumes"
Oct 31 03:59:31.683633 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:31.683595    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"]
Oct 31 03:59:36.684412 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:36.684368    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp" podUID="c0535859-46d6-489e-b7a4-8601db014cfb" containerName="payload-pre-processing" containerID="cri-o://04463094e8dc6fb1644f8dc13dac509cc24b4aa0474f8729e40d5f4faa426a21" gracePeriod=25
Oct 31 03:59:36.960986 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:36.960953    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"
Oct 31 03:59:37.076281 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.076249    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0535859-46d6-489e-b7a4-8601db014cfb-kube-api-access-pdtzp\" (UniqueName: \"kubernetes.io/projected/c0535859-46d6-489e-b7a4-8601db014cfb-kube-api-access-pdtzp\") pod \"c0535859-46d6-489e-b7a4-8601db014cfb\" (UID: \"c0535859-46d6-489e-b7a4-8601db014cfb\") "
Oct 31 03:59:37.076481 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.076371    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0535859-46d6-489e-b7a4-8601db014cfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0535859-46d6-489e-b7a4-8601db014cfb-plugins-config-volume\") pod \"c0535859-46d6-489e-b7a4-8601db014cfb\" (UID: \"c0535859-46d6-489e-b7a4-8601db014cfb\") "
Oct 31 03:59:37.076815 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.076791    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0535859-46d6-489e-b7a4-8601db014cfb-plugins-config-volume" pod "c0535859-46d6-489e-b7a4-8601db014cfb" (UID: "c0535859-46d6-489e-b7a4-8601db014cfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:37.078702 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.078679    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0535859-46d6-489e-b7a4-8601db014cfb-kube-api-access-pdtzp" pod "c0535859-46d6-489e-b7a4-8601db014cfb" (UID: "c0535859-46d6-489e-b7a4-8601db014cfb"). InnerVolumeSpecName "kube-api-access-pdtzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:37.182012 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.181968    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0535859-46d6-489e-b7a4-8601db014cfb-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:37.182012 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.182011    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdtzp\" (UniqueName: \"kubernetes.io/projected/c0535859-46d6-489e-b7a4-8601db014cfb-kube-api-access-pdtzp\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:37.376373 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.376335    5415 generic.go:356] "Generic (PLEG): container finished" podID="c0535859-46d6-489e-b7a4-8601db014cfb" containerID="04463094e8dc6fb1644f8dc13dac509cc24b4aa0474f8729e40d5f4faa426a21" exitCode=0
Oct 31 03:59:37.376577 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.376378    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp" event={"ID":"c0535859-46d6-489e-b7a4-8601db014cfb","Type":"ContainerDied","Data":"04463094e8dc6fb1644f8dc13dac509cc24b4aa0474f8729e40d5f4faa426a21"}
Oct 31 03:59:37.376577 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.376407    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"
Oct 31 03:59:37.376577 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.376451    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp" event={"ID":"c0535859-46d6-489e-b7a4-8601db014cfb","Type":"ContainerDied","Data":"ade285edc5a97d514aa6dfcd3eeee5bdbbc0de4ca0dd78149d8e8a2d3170c1af"}
Oct 31 03:59:37.376577 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.376487    5415 scope.go:122] "RemoveContainer" containerID="04463094e8dc6fb1644f8dc13dac509cc24b4aa0474f8729e40d5f4faa426a21"
Oct 31 03:59:37.386844 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.386825    5415 scope.go:122] "RemoveContainer" containerID="04463094e8dc6fb1644f8dc13dac509cc24b4aa0474f8729e40d5f4faa426a21"
Oct 31 03:59:37.387208 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:59:37.387187    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04463094e8dc6fb1644f8dc13dac509cc24b4aa0474f8729e40d5f4faa426a21\": container with ID starting with 04463094e8dc6fb1644f8dc13dac509cc24b4aa0474f8729e40d5f4faa426a21 not found: ID does not exist" containerID="04463094e8dc6fb1644f8dc13dac509cc24b4aa0474f8729e40d5f4faa426a21"
Oct 31 03:59:37.387300 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.387224    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04463094e8dc6fb1644f8dc13dac509cc24b4aa0474f8729e40d5f4faa426a21"} err="failed to get container status \"04463094e8dc6fb1644f8dc13dac509cc24b4aa0474f8729e40d5f4faa426a21\": rpc error: code = NotFound desc = could not find container \"04463094e8dc6fb1644f8dc13dac509cc24b4aa0474f8729e40d5f4faa426a21\": container with ID starting with 04463094e8dc6fb1644f8dc13dac509cc24b4aa0474f8729e40d5f4faa426a21 not found: ID does not exist"
Oct 31 03:59:37.532829 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.532798    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"]
Oct 31 03:59:37.536616 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:37.536585    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dffe64-7c97ff4879-mmcqp"]
Oct 31 03:59:38.607176 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:38.607142    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0535859-46d6-489e-b7a4-8601db014cfb" path="/var/lib/kubelet/pods/c0535859-46d6-489e-b7a4-8601db014cfb/volumes"
Oct 31 03:59:44.252131 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.252095    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"]
Oct 31 03:59:44.252562 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.252321    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs" podUID="426d21ff-8092-4c91-85f1-76bdcd3e58b8" containerName="maas-api" containerID="cri-o://2c9fae770599826ce447fdf4aec5aa96dd9db190eb2180c4815cf358b57c1ca0" gracePeriod=30
Oct 31 03:59:44.423552 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.423518    5415 generic.go:356] "Generic (PLEG): container finished" podID="426d21ff-8092-4c91-85f1-76bdcd3e58b8" containerID="2c9fae770599826ce447fdf4aec5aa96dd9db190eb2180c4815cf358b57c1ca0" exitCode=0
Oct 31 03:59:44.423741 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.423643    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs" event={"ID":"426d21ff-8092-4c91-85f1-76bdcd3e58b8","Type":"ContainerDied","Data":"2c9fae770599826ce447fdf4aec5aa96dd9db190eb2180c4815cf358b57c1ca0"}
Oct 31 03:59:44.541264 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.541230    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:59:44.582146 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.582108    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426d21ff-8092-4c91-85f1-76bdcd3e58b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d21ff-8092-4c91-85f1-76bdcd3e58b8-openshift-service-ca\") pod \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") "
Oct 31 03:59:44.582365 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.582169    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-maas-api-tls\") pod \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") "
Oct 31 03:59:44.582365 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.582230    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426d21ff-8092-4c91-85f1-76bdcd3e58b8-kube-api-access-xctgw\" (UniqueName: \"kubernetes.io/projected/426d21ff-8092-4c91-85f1-76bdcd3e58b8-kube-api-access-xctgw\") pod \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") "
Oct 31 03:59:44.582365 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.582329    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-metrics-certs\") pod \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\" (UID: \"426d21ff-8092-4c91-85f1-76bdcd3e58b8\") "
Oct 31 03:59:44.582577 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.582536    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426d21ff-8092-4c91-85f1-76bdcd3e58b8-openshift-service-ca" pod "426d21ff-8092-4c91-85f1-76bdcd3e58b8" (UID: "426d21ff-8092-4c91-85f1-76bdcd3e58b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:44.582989 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.582902    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d21ff-8092-4c91-85f1-76bdcd3e58b8-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:44.586274 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.585264    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-metrics-certs" pod "426d21ff-8092-4c91-85f1-76bdcd3e58b8" (UID: "426d21ff-8092-4c91-85f1-76bdcd3e58b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:44.586274 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.585283    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-maas-api-tls" pod "426d21ff-8092-4c91-85f1-76bdcd3e58b8" (UID: "426d21ff-8092-4c91-85f1-76bdcd3e58b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:44.586274 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.585541    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426d21ff-8092-4c91-85f1-76bdcd3e58b8-kube-api-access-xctgw" pod "426d21ff-8092-4c91-85f1-76bdcd3e58b8" (UID: "426d21ff-8092-4c91-85f1-76bdcd3e58b8"). InnerVolumeSpecName "kube-api-access-xctgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:44.684818 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.684784    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:44.684818 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.684818    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/426d21ff-8092-4c91-85f1-76bdcd3e58b8-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:44.685128 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:44.684831    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xctgw\" (UniqueName: \"kubernetes.io/projected/426d21ff-8092-4c91-85f1-76bdcd3e58b8-kube-api-access-xctgw\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:45.430503 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:45.430475    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"
Oct 31 03:59:45.430503 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:45.430489    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs" event={"ID":"426d21ff-8092-4c91-85f1-76bdcd3e58b8","Type":"ContainerDied","Data":"4a92baab2538cb096d2b6c41b61d86bc55f900dad9f41c26a9a2197368f9ed86"}
Oct 31 03:59:45.431035 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:45.430549    5415 scope.go:122] "RemoveContainer" containerID="2c9fae770599826ce447fdf4aec5aa96dd9db190eb2180c4815cf358b57c1ca0"
Oct 31 03:59:45.587900 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:45.587779    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"]
Oct 31 03:59:45.591432 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:45.591398    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dffe64-b57db78b5-t8rxs"]
Oct 31 03:59:46.177074 ip-10-0-139-164 kubenswrapper[5415]: E1031 03:59:46.177032    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 03:59:46.606807 ip-10-0-139-164 kubenswrapper[5415]: I1031 03:59:46.606770    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426d21ff-8092-4c91-85f1-76bdcd3e58b8" path="/var/lib/kubelet/pods/426d21ff-8092-4c91-85f1-76bdcd3e58b8/volumes"
Oct 31 04:00:00.253097 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253060    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"]
Oct 31 04:00:00.253628 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253614    5415 cpu_manager.go:414] "containerMap: removing container" podUID="426d21ff-8092-4c91-85f1-76bdcd3e58b8" containerName="maas-api"
Oct 31 04:00:00.253628 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253629    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426d21ff-8092-4c91-85f1-76bdcd3e58b8" containerName="maas-api"
Oct 31 04:00:00.253709 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253657    5415 cpu_manager.go:414] "containerMap: removing container" podUID="c0535859-46d6-489e-b7a4-8601db014cfb" containerName="payload-pre-processing"
Oct 31 04:00:00.253709 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253662    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0535859-46d6-489e-b7a4-8601db014cfb" containerName="payload-pre-processing"
Oct 31 04:00:00.253709 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253675    5415 cpu_manager.go:414] "containerMap: removing container" podUID="19da59fb-854b-4e33-b635-c68551bf4179" containerName="istio-proxy"
Oct 31 04:00:00.253709 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253679    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19da59fb-854b-4e33-b635-c68551bf4179" containerName="istio-proxy"
Oct 31 04:00:00.253709 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253697    5415 cpu_manager.go:414] "containerMap: removing container" podUID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" containerName="istio-proxy"
Oct 31 04:00:00.253709 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253702    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" containerName="istio-proxy"
Oct 31 04:00:00.253709 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253710    5415 cpu_manager.go:414] "containerMap: removing container" podUID="c64a42a6-ace4-42e2-bf65-f834e9c4c263" containerName="istio-proxy"
Oct 31 04:00:00.253944 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253715    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64a42a6-ace4-42e2-bf65-f834e9c4c263" containerName="istio-proxy"
Oct 31 04:00:00.253944 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253786    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64a42a6-ace4-42e2-bf65-f834e9c4c263" containerName="istio-proxy"
Oct 31 04:00:00.253944 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253796    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="426d21ff-8092-4c91-85f1-76bdcd3e58b8" containerName="maas-api"
Oct 31 04:00:00.253944 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253805    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0535859-46d6-489e-b7a4-8601db014cfb" containerName="payload-pre-processing"
Oct 31 04:00:00.253944 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253811    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="19da59fb-854b-4e33-b635-c68551bf4179" containerName="istio-proxy"
Oct 31 04:00:00.253944 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.253820    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="f707fded-4ede-4eb3-99a2-9c16ccb99f50" containerName="istio-proxy"
Oct 31 04:00:00.258552 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.258532    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"
Oct 31 04:00:00.268694 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.268663    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"]
Oct 31 04:00:00.272666 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.272647    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"
Oct 31 04:00:00.281403 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.281373    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"]
Oct 31 04:00:00.286167 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.286148    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"
Oct 31 04:00:00.340521 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.340438    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"]
Oct 31 04:00:00.363374 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.363342    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:00:00.373471 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.373432    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwx48\" (UniqueName: \"kubernetes.io/projected/8d03d8d7-4995-493f-93b5-0a601eccc765-kube-api-access-xwx48\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85\" (UID: \"8d03d8d7-4995-493f-93b5-0a601eccc765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"
Oct 31 04:00:00.373658 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.373593    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a050ec-c2ba-4bf4-a625-36ce24f78a62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw\" (UID: \"22a050ec-c2ba-4bf4-a625-36ce24f78a62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"
Oct 31 04:00:00.373658 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.373641    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b436852d-3c87-49f4-9eb4-c66586dd1fd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj\" (UID: \"b436852d-3c87-49f4-9eb4-c66586dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"
Oct 31 04:00:00.373831 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.373805    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d03d8d7-4995-493f-93b5-0a601eccc765-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85\" (UID: \"8d03d8d7-4995-493f-93b5-0a601eccc765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"
Oct 31 04:00:00.373902 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.373847    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5lk9\" (UniqueName: \"kubernetes.io/projected/b436852d-3c87-49f4-9eb4-c66586dd1fd2-kube-api-access-h5lk9\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj\" (UID: \"b436852d-3c87-49f4-9eb4-c66586dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"
Oct 31 04:00:00.373945 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.373923    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vmgh\" (UniqueName: \"kubernetes.io/projected/22a050ec-c2ba-4bf4-a625-36ce24f78a62-kube-api-access-4vmgh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw\" (UID: \"22a050ec-c2ba-4bf4-a625-36ce24f78a62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"
Oct 31 04:00:00.388983 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.388948    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"]
Oct 31 04:00:00.413531 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.413495    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"]
Oct 31 04:00:00.475557 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.475520    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5lk9\" (UniqueName: \"kubernetes.io/projected/b436852d-3c87-49f4-9eb4-c66586dd1fd2-kube-api-access-h5lk9\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj\" (UID: \"b436852d-3c87-49f4-9eb4-c66586dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"
Oct 31 04:00:00.475744 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.475567    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vmgh\" (UniqueName: \"kubernetes.io/projected/22a050ec-c2ba-4bf4-a625-36ce24f78a62-kube-api-access-4vmgh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw\" (UID: \"22a050ec-c2ba-4bf4-a625-36ce24f78a62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"
Oct 31 04:00:00.475744 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.475618    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwx48\" (UniqueName: \"kubernetes.io/projected/8d03d8d7-4995-493f-93b5-0a601eccc765-kube-api-access-xwx48\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85\" (UID: \"8d03d8d7-4995-493f-93b5-0a601eccc765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"
Oct 31 04:00:00.475744 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.475690    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a050ec-c2ba-4bf4-a625-36ce24f78a62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw\" (UID: \"22a050ec-c2ba-4bf4-a625-36ce24f78a62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"
Oct 31 04:00:00.475744 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.475724    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b436852d-3c87-49f4-9eb4-c66586dd1fd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj\" (UID: \"b436852d-3c87-49f4-9eb4-c66586dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"
Oct 31 04:00:00.475939 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.475790    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d03d8d7-4995-493f-93b5-0a601eccc765-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85\" (UID: \"8d03d8d7-4995-493f-93b5-0a601eccc765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"
Oct 31 04:00:00.476436 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.476417    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d03d8d7-4995-493f-93b5-0a601eccc765-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85\" (UID: \"8d03d8d7-4995-493f-93b5-0a601eccc765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"
Oct 31 04:00:00.476571 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.476550    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b436852d-3c87-49f4-9eb4-c66586dd1fd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj\" (UID: \"b436852d-3c87-49f4-9eb4-c66586dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"
Oct 31 04:00:00.476609 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.476550    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a050ec-c2ba-4bf4-a625-36ce24f78a62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw\" (UID: \"22a050ec-c2ba-4bf4-a625-36ce24f78a62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"
Oct 31 04:00:00.530156 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.530124    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vmgh\" (UniqueName: \"kubernetes.io/projected/22a050ec-c2ba-4bf4-a625-36ce24f78a62-kube-api-access-4vmgh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw\" (UID: \"22a050ec-c2ba-4bf4-a625-36ce24f78a62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"
Oct 31 04:00:00.530773 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.530745    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwx48\" (UniqueName: \"kubernetes.io/projected/8d03d8d7-4995-493f-93b5-0a601eccc765-kube-api-access-xwx48\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85\" (UID: \"8d03d8d7-4995-493f-93b5-0a601eccc765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"
Oct 31 04:00:00.531075 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.531059    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5lk9\" (UniqueName: \"kubernetes.io/projected/b436852d-3c87-49f4-9eb4-c66586dd1fd2-kube-api-access-h5lk9\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj\" (UID: \"b436852d-3c87-49f4-9eb4-c66586dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"
Oct 31 04:00:00.574041 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.573997    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"
Oct 31 04:00:00.588447 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.588422    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"
Oct 31 04:00:00.598780 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.598749    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"
Oct 31 04:00:00.961447 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:00.961413    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"]
Oct 31 04:00:00.964726 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:00:00.964692    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d03d8d7_4995_493f_93b5_0a601eccc765.slice/crio-14d44ffe553f654e08e513ea25ff0d5d97b7d43c2b2e4fa1320d78d89462aa7d WatchSource:0}: Error finding container 14d44ffe553f654e08e513ea25ff0d5d97b7d43c2b2e4fa1320d78d89462aa7d: Status 404 returned error can't find the container with id 14d44ffe553f654e08e513ea25ff0d5d97b7d43c2b2e4fa1320d78d89462aa7d
Oct 31 04:00:01.008030 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:01.008005    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"]
Oct 31 04:00:01.011125 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:00:01.011093    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb436852d_3c87_49f4_9eb4_c66586dd1fd2.slice/crio-95e7a6b6edf8dcc5ecd192c6a04bacbe0eedabc22b8b36bb87dc38f146ad2e44 WatchSource:0}: Error finding container 95e7a6b6edf8dcc5ecd192c6a04bacbe0eedabc22b8b36bb87dc38f146ad2e44: Status 404 returned error can't find the container with id 95e7a6b6edf8dcc5ecd192c6a04bacbe0eedabc22b8b36bb87dc38f146ad2e44
Oct 31 04:00:01.052548 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:01.052524    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"]
Oct 31 04:00:01.150253 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:00:01.150219    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a050ec_c2ba_4bf4_a625_36ce24f78a62.slice/crio-ca78f24fb80e160837e7a3ecc103390a7c16b07afbfd95c1375d1227e4146d86 WatchSource:0}: Error finding container ca78f24fb80e160837e7a3ecc103390a7c16b07afbfd95c1375d1227e4146d86: Status 404 returned error can't find the container with id ca78f24fb80e160837e7a3ecc103390a7c16b07afbfd95c1375d1227e4146d86
Oct 31 04:00:01.528754 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:01.528716    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw" event={"ID":"22a050ec-c2ba-4bf4-a625-36ce24f78a62","Type":"ContainerStarted","Data":"fe24899313e8105a3105454906941816321f14e85ea234c73f97d6ad504cb0d3"}
Oct 31 04:00:01.529224 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:01.528777    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw" event={"ID":"22a050ec-c2ba-4bf4-a625-36ce24f78a62","Type":"ContainerStarted","Data":"ca78f24fb80e160837e7a3ecc103390a7c16b07afbfd95c1375d1227e4146d86"}
Oct 31 04:00:01.530727 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:01.530696    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85" event={"ID":"8d03d8d7-4995-493f-93b5-0a601eccc765","Type":"ContainerStarted","Data":"ca6a7c9b14e3589c5f73f5bf8e8e244f668d1ad7642a99c948abb35f163b4498"}
Oct 31 04:00:01.530873 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:01.530750    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85" event={"ID":"8d03d8d7-4995-493f-93b5-0a601eccc765","Type":"ContainerStarted","Data":"14d44ffe553f654e08e513ea25ff0d5d97b7d43c2b2e4fa1320d78d89462aa7d"}
Oct 31 04:00:01.532511 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:01.532482    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj" event={"ID":"b436852d-3c87-49f4-9eb4-c66586dd1fd2","Type":"ContainerStarted","Data":"35c94edd6eabce277ab47a2ec9be81dbdec5ee899dc148795b4fbff8c034c5bd"}
Oct 31 04:00:01.532613 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:01.532531    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj" event={"ID":"b436852d-3c87-49f4-9eb4-c66586dd1fd2","Type":"ContainerStarted","Data":"95e7a6b6edf8dcc5ecd192c6a04bacbe0eedabc22b8b36bb87dc38f146ad2e44"}
Oct 31 04:00:01.639456 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:01.639353    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw" podStartSLOduration=1.639338454 podStartE2EDuration="1.639338454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:01.63843348 +0000 UTC m=+3143.198913231" watchObservedRunningTime="2026-10-31 04:00:01.639338454 +0000 UTC m=+3143.199818185"
Oct 31 04:00:01.752569 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:01.752521    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85" podStartSLOduration=1.7525073340000001 podStartE2EDuration="1.752507334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:01.749692572 +0000 UTC m=+3143.310172306" watchObservedRunningTime="2026-10-31 04:00:01.752507334 +0000 UTC m=+3143.312987063"
Oct 31 04:00:01.859015 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:01.858963    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj" podStartSLOduration=1.858946536 podStartE2EDuration="1.858946536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:01.856447398 +0000 UTC m=+3143.416927146" watchObservedRunningTime="2026-10-31 04:00:01.858946536 +0000 UTC m=+3143.419426266"
Oct 31 04:00:02.542800 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:02.542761    5415 generic.go:356] "Generic (PLEG): container finished" podID="b436852d-3c87-49f4-9eb4-c66586dd1fd2" containerID="35c94edd6eabce277ab47a2ec9be81dbdec5ee899dc148795b4fbff8c034c5bd" exitCode=60
Oct 31 04:00:02.543302 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:02.542824    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj" event={"ID":"b436852d-3c87-49f4-9eb4-c66586dd1fd2","Type":"ContainerDied","Data":"35c94edd6eabce277ab47a2ec9be81dbdec5ee899dc148795b4fbff8c034c5bd"}
Oct 31 04:00:02.543302 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:02.543202    5415 scope.go:122] "RemoveContainer" containerID="35c94edd6eabce277ab47a2ec9be81dbdec5ee899dc148795b4fbff8c034c5bd"
Oct 31 04:00:02.545318 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:02.545294    5415 generic.go:356] "Generic (PLEG): container finished" podID="22a050ec-c2ba-4bf4-a625-36ce24f78a62" containerID="fe24899313e8105a3105454906941816321f14e85ea234c73f97d6ad504cb0d3" exitCode=60
Oct 31 04:00:02.545560 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:02.545538    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw" event={"ID":"22a050ec-c2ba-4bf4-a625-36ce24f78a62","Type":"ContainerDied","Data":"fe24899313e8105a3105454906941816321f14e85ea234c73f97d6ad504cb0d3"}
Oct 31 04:00:02.545982 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:02.545954    5415 scope.go:122] "RemoveContainer" containerID="fe24899313e8105a3105454906941816321f14e85ea234c73f97d6ad504cb0d3"
Oct 31 04:00:02.548080 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:02.548058    5415 generic.go:356] "Generic (PLEG): container finished" podID="8d03d8d7-4995-493f-93b5-0a601eccc765" containerID="ca6a7c9b14e3589c5f73f5bf8e8e244f668d1ad7642a99c948abb35f163b4498" exitCode=60
Oct 31 04:00:02.548152 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:02.548093    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85" event={"ID":"8d03d8d7-4995-493f-93b5-0a601eccc765","Type":"ContainerDied","Data":"ca6a7c9b14e3589c5f73f5bf8e8e244f668d1ad7642a99c948abb35f163b4498"}
Oct 31 04:00:02.548492 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:02.548474    5415 scope.go:122] "RemoveContainer" containerID="ca6a7c9b14e3589c5f73f5bf8e8e244f668d1ad7642a99c948abb35f163b4498"
Oct 31 04:00:03.556441 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:03.556403    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85" event={"ID":"8d03d8d7-4995-493f-93b5-0a601eccc765","Type":"ContainerStarted","Data":"e3c44c8781bea3331e134cdb4b6cf69971daea51ddb0848c202e0a587e221b34"}
Oct 31 04:00:03.558439 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:03.558412    5415 generic.go:356] "Generic (PLEG): container finished" podID="b436852d-3c87-49f4-9eb4-c66586dd1fd2" containerID="f5beff09857b4ad16f388a3dd6598aaa3e47b83716e028e1ba8d19673b3950e4" exitCode=60
Oct 31 04:00:03.558560 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:03.558495    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj" event={"ID":"b436852d-3c87-49f4-9eb4-c66586dd1fd2","Type":"ContainerDied","Data":"f5beff09857b4ad16f388a3dd6598aaa3e47b83716e028e1ba8d19673b3950e4"}
Oct 31 04:00:03.558635 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:03.558559    5415 scope.go:122] "RemoveContainer" containerID="35c94edd6eabce277ab47a2ec9be81dbdec5ee899dc148795b4fbff8c034c5bd"
Oct 31 04:00:03.558845 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:03.558822    5415 scope.go:122] "RemoveContainer" containerID="f5beff09857b4ad16f388a3dd6598aaa3e47b83716e028e1ba8d19673b3950e4"
Oct 31 04:00:03.559237 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:00:03.559214    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj_odh-ai-gateway-infra(b436852d-3c87-49f4-9eb4-c66586dd1fd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj" podUID="b436852d-3c87-49f4-9eb4-c66586dd1fd2"
Oct 31 04:00:03.560588 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:03.560566    5415 generic.go:356] "Generic (PLEG): container finished" podID="22a050ec-c2ba-4bf4-a625-36ce24f78a62" containerID="026f64c717474e576823b2b17c0debe1de9a1e1b07dd3ff7e4542a8cfdcb8e5b" exitCode=60
Oct 31 04:00:03.560682 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:03.560620    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw" event={"ID":"22a050ec-c2ba-4bf4-a625-36ce24f78a62","Type":"ContainerDied","Data":"026f64c717474e576823b2b17c0debe1de9a1e1b07dd3ff7e4542a8cfdcb8e5b"}
Oct 31 04:00:03.560863 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:03.560849    5415 scope.go:122] "RemoveContainer" containerID="026f64c717474e576823b2b17c0debe1de9a1e1b07dd3ff7e4542a8cfdcb8e5b"
Oct 31 04:00:03.561119 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:00:03.561102    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw_odh-ai-gateway-infra(22a050ec-c2ba-4bf4-a625-36ce24f78a62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw" podUID="22a050ec-c2ba-4bf4-a625-36ce24f78a62"
Oct 31 04:00:03.628714 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:03.628686    5415 scope.go:122] "RemoveContainer" containerID="fe24899313e8105a3105454906941816321f14e85ea234c73f97d6ad504cb0d3"
Oct 31 04:00:04.568240 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:04.568207    5415 scope.go:122] "RemoveContainer" containerID="026f64c717474e576823b2b17c0debe1de9a1e1b07dd3ff7e4542a8cfdcb8e5b"
Oct 31 04:00:04.568746 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:00:04.568522    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw_odh-ai-gateway-infra(22a050ec-c2ba-4bf4-a625-36ce24f78a62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw" podUID="22a050ec-c2ba-4bf4-a625-36ce24f78a62"
Oct 31 04:00:04.570000 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:04.569970    5415 generic.go:356] "Generic (PLEG): container finished" podID="8d03d8d7-4995-493f-93b5-0a601eccc765" containerID="e3c44c8781bea3331e134cdb4b6cf69971daea51ddb0848c202e0a587e221b34" exitCode=60
Oct 31 04:00:04.570138 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:04.570065    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85" event={"ID":"8d03d8d7-4995-493f-93b5-0a601eccc765","Type":"ContainerDied","Data":"e3c44c8781bea3331e134cdb4b6cf69971daea51ddb0848c202e0a587e221b34"}
Oct 31 04:00:04.570204 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:04.570135    5415 scope.go:122] "RemoveContainer" containerID="ca6a7c9b14e3589c5f73f5bf8e8e244f668d1ad7642a99c948abb35f163b4498"
Oct 31 04:00:04.570304 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:04.570290    5415 scope.go:122] "RemoveContainer" containerID="e3c44c8781bea3331e134cdb4b6cf69971daea51ddb0848c202e0a587e221b34"
Oct 31 04:00:04.570609 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:00:04.570577    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85_odh-ai-gateway-infra(8d03d8d7-4995-493f-93b5-0a601eccc765)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85" podUID="8d03d8d7-4995-493f-93b5-0a601eccc765"
Oct 31 04:00:04.571987 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:04.571968    5415 scope.go:122] "RemoveContainer" containerID="f5beff09857b4ad16f388a3dd6598aaa3e47b83716e028e1ba8d19673b3950e4"
Oct 31 04:00:04.572226 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:00:04.572210    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj_odh-ai-gateway-infra(b436852d-3c87-49f4-9eb4-c66586dd1fd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj" podUID="b436852d-3c87-49f4-9eb4-c66586dd1fd2"
Oct 31 04:00:05.578984 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:05.578956    5415 scope.go:122] "RemoveContainer" containerID="e3c44c8781bea3331e134cdb4b6cf69971daea51ddb0848c202e0a587e221b34"
Oct 31 04:00:05.579396 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:00:05.579229    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85_odh-ai-gateway-infra(8d03d8d7-4995-493f-93b5-0a601eccc765)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85" podUID="8d03d8d7-4995-493f-93b5-0a601eccc765"
Oct 31 04:00:15.599533 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:15.599455    5415 scope.go:122] "RemoveContainer" containerID="e3c44c8781bea3331e134cdb4b6cf69971daea51ddb0848c202e0a587e221b34"
Oct 31 04:00:15.599533 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:15.599524    5415 scope.go:122] "RemoveContainer" containerID="026f64c717474e576823b2b17c0debe1de9a1e1b07dd3ff7e4542a8cfdcb8e5b"
Oct 31 04:00:16.176862 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:00:16.176782    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:00:16.643567 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:16.643534    5415 generic.go:356] "Generic (PLEG): container finished" podID="8d03d8d7-4995-493f-93b5-0a601eccc765" containerID="49a4671f576a87ed983e2106dd549cfd96973d81964d33cd88e21fa2a82827c6" exitCode=60
Oct 31 04:00:16.644059 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:16.643607    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85" event={"ID":"8d03d8d7-4995-493f-93b5-0a601eccc765","Type":"ContainerDied","Data":"49a4671f576a87ed983e2106dd549cfd96973d81964d33cd88e21fa2a82827c6"}
Oct 31 04:00:16.644059 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:16.643672    5415 scope.go:122] "RemoveContainer" containerID="e3c44c8781bea3331e134cdb4b6cf69971daea51ddb0848c202e0a587e221b34"
Oct 31 04:00:16.644182 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:16.644108    5415 scope.go:122] "RemoveContainer" containerID="49a4671f576a87ed983e2106dd549cfd96973d81964d33cd88e21fa2a82827c6"
Oct 31 04:00:16.644407 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:00:16.644386    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85_odh-ai-gateway-infra(8d03d8d7-4995-493f-93b5-0a601eccc765)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85" podUID="8d03d8d7-4995-493f-93b5-0a601eccc765"
Oct 31 04:00:16.646017 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:16.645996    5415 generic.go:356] "Generic (PLEG): container finished" podID="22a050ec-c2ba-4bf4-a625-36ce24f78a62" containerID="37cd466cd47c49e2f192ab43d2cbd4d2880634f15511fc603d75cf0f3e120858" exitCode=60
Oct 31 04:00:16.646092 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:16.646064    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw" event={"ID":"22a050ec-c2ba-4bf4-a625-36ce24f78a62","Type":"ContainerDied","Data":"37cd466cd47c49e2f192ab43d2cbd4d2880634f15511fc603d75cf0f3e120858"}
Oct 31 04:00:16.646408 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:16.646395    5415 scope.go:122] "RemoveContainer" containerID="37cd466cd47c49e2f192ab43d2cbd4d2880634f15511fc603d75cf0f3e120858"
Oct 31 04:00:16.646658 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:00:16.646643    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw_odh-ai-gateway-infra(22a050ec-c2ba-4bf4-a625-36ce24f78a62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw" podUID="22a050ec-c2ba-4bf4-a625-36ce24f78a62"
Oct 31 04:00:16.702538 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:16.702513    5415 scope.go:122] "RemoveContainer" containerID="026f64c717474e576823b2b17c0debe1de9a1e1b07dd3ff7e4542a8cfdcb8e5b"
Oct 31 04:00:17.599300 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:17.599270    5415 scope.go:122] "RemoveContainer" containerID="f5beff09857b4ad16f388a3dd6598aaa3e47b83716e028e1ba8d19673b3950e4"
Oct 31 04:00:17.765498 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:17.765461    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"]
Oct 31 04:00:17.873048 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:17.873013    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"]
Oct 31 04:00:17.986146 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:17.986119    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"
Oct 31 04:00:18.097330 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.097301    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d03d8d7-4995-493f-93b5-0a601eccc765-kube-api-access-xwx48\" (UniqueName: \"kubernetes.io/projected/8d03d8d7-4995-493f-93b5-0a601eccc765-kube-api-access-xwx48\") pod \"8d03d8d7-4995-493f-93b5-0a601eccc765\" (UID: \"8d03d8d7-4995-493f-93b5-0a601eccc765\") "
Oct 31 04:00:18.097497 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.097372    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d03d8d7-4995-493f-93b5-0a601eccc765-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d03d8d7-4995-493f-93b5-0a601eccc765-maas-api-service-ca\") pod \"8d03d8d7-4995-493f-93b5-0a601eccc765\" (UID: \"8d03d8d7-4995-493f-93b5-0a601eccc765\") "
Oct 31 04:00:18.097796 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.097766    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d03d8d7-4995-493f-93b5-0a601eccc765-maas-api-service-ca" pod "8d03d8d7-4995-493f-93b5-0a601eccc765" (UID: "8d03d8d7-4995-493f-93b5-0a601eccc765"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:18.097977 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.097951    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d03d8d7-4995-493f-93b5-0a601eccc765-maas-api-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:18.099619 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.099473    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d03d8d7-4995-493f-93b5-0a601eccc765-kube-api-access-xwx48" pod "8d03d8d7-4995-493f-93b5-0a601eccc765" (UID: "8d03d8d7-4995-493f-93b5-0a601eccc765"). InnerVolumeSpecName "kube-api-access-xwx48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:18.131513 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.131483    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"
Oct 31 04:00:18.198431 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.198397    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a050ec-c2ba-4bf4-a625-36ce24f78a62-kube-api-access-4vmgh\" (UniqueName: \"kubernetes.io/projected/22a050ec-c2ba-4bf4-a625-36ce24f78a62-kube-api-access-4vmgh\") pod \"22a050ec-c2ba-4bf4-a625-36ce24f78a62\" (UID: \"22a050ec-c2ba-4bf4-a625-36ce24f78a62\") "
Oct 31 04:00:18.198630 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.198521    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a050ec-c2ba-4bf4-a625-36ce24f78a62-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a050ec-c2ba-4bf4-a625-36ce24f78a62-maas-api-service-ca\") pod \"22a050ec-c2ba-4bf4-a625-36ce24f78a62\" (UID: \"22a050ec-c2ba-4bf4-a625-36ce24f78a62\") "
Oct 31 04:00:18.199008 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.198979    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a050ec-c2ba-4bf4-a625-36ce24f78a62-maas-api-service-ca" pod "22a050ec-c2ba-4bf4-a625-36ce24f78a62" (UID: "22a050ec-c2ba-4bf4-a625-36ce24f78a62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:18.199221 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.199201    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a050ec-c2ba-4bf4-a625-36ce24f78a62-maas-api-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:18.199266 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.199227    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwx48\" (UniqueName: \"kubernetes.io/projected/8d03d8d7-4995-493f-93b5-0a601eccc765-kube-api-access-xwx48\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:18.200646 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.200624    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a050ec-c2ba-4bf4-a625-36ce24f78a62-kube-api-access-4vmgh" pod "22a050ec-c2ba-4bf4-a625-36ce24f78a62" (UID: "22a050ec-c2ba-4bf4-a625-36ce24f78a62"). InnerVolumeSpecName "kube-api-access-4vmgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:18.300344 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.300309    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vmgh\" (UniqueName: \"kubernetes.io/projected/22a050ec-c2ba-4bf4-a625-36ce24f78a62-kube-api-access-4vmgh\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:18.665326 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.665288    5415 generic.go:356] "Generic (PLEG): container finished" podID="b436852d-3c87-49f4-9eb4-c66586dd1fd2" containerID="25c91d0a584f411c7c979b7a656bac5437f2f6adac709cd2dc779d43c1b5ee28" exitCode=60
Oct 31 04:00:18.665525 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.665360    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj" event={"ID":"b436852d-3c87-49f4-9eb4-c66586dd1fd2","Type":"ContainerDied","Data":"25c91d0a584f411c7c979b7a656bac5437f2f6adac709cd2dc779d43c1b5ee28"}
Oct 31 04:00:18.665525 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.665419    5415 scope.go:122] "RemoveContainer" containerID="f5beff09857b4ad16f388a3dd6598aaa3e47b83716e028e1ba8d19673b3950e4"
Oct 31 04:00:18.665815 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.665797    5415 scope.go:122] "RemoveContainer" containerID="25c91d0a584f411c7c979b7a656bac5437f2f6adac709cd2dc779d43c1b5ee28"
Oct 31 04:00:18.666176 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:00:18.666143    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj_odh-ai-gateway-infra(b436852d-3c87-49f4-9eb4-c66586dd1fd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj" podUID="b436852d-3c87-49f4-9eb4-c66586dd1fd2"
Oct 31 04:00:18.666854 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.666838    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"
Oct 31 04:00:18.666949 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.666838    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw" event={"ID":"22a050ec-c2ba-4bf4-a625-36ce24f78a62","Type":"ContainerDied","Data":"ca78f24fb80e160837e7a3ecc103390a7c16b07afbfd95c1375d1227e4146d86"}
Oct 31 04:00:18.668411 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.668381    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85" event={"ID":"8d03d8d7-4995-493f-93b5-0a601eccc765","Type":"ContainerDied","Data":"14d44ffe553f654e08e513ea25ff0d5d97b7d43c2b2e4fa1320d78d89462aa7d"}
Oct 31 04:00:18.668515 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.668428    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"
Oct 31 04:00:18.681786 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.681761    5415 scope.go:122] "RemoveContainer" containerID="37cd466cd47c49e2f192ab43d2cbd4d2880634f15511fc603d75cf0f3e120858"
Oct 31 04:00:18.690531 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.690509    5415 scope.go:122] "RemoveContainer" containerID="49a4671f576a87ed983e2106dd549cfd96973d81964d33cd88e21fa2a82827c6"
Oct 31 04:00:18.879978 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.879933    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"]
Oct 31 04:00:18.883236 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.883209    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890320-64h85"]
Oct 31 04:00:18.985778 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.985743    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"]
Oct 31 04:00:18.989290 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:18.989264    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890320-dccrw"]
Oct 31 04:00:19.785097 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:19.785060    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"]
Oct 31 04:00:19.953795 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:19.953767    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"
Oct 31 04:00:20.021094 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:20.021060    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b436852d-3c87-49f4-9eb4-c66586dd1fd2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b436852d-3c87-49f4-9eb4-c66586dd1fd2-maas-api-service-ca\") pod \"b436852d-3c87-49f4-9eb4-c66586dd1fd2\" (UID: \"b436852d-3c87-49f4-9eb4-c66586dd1fd2\") "
Oct 31 04:00:20.021302 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:20.021228    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b436852d-3c87-49f4-9eb4-c66586dd1fd2-kube-api-access-h5lk9\" (UniqueName: \"kubernetes.io/projected/b436852d-3c87-49f4-9eb4-c66586dd1fd2-kube-api-access-h5lk9\") pod \"b436852d-3c87-49f4-9eb4-c66586dd1fd2\" (UID: \"b436852d-3c87-49f4-9eb4-c66586dd1fd2\") "
Oct 31 04:00:20.021510 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:20.021487    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b436852d-3c87-49f4-9eb4-c66586dd1fd2-maas-api-service-ca" pod "b436852d-3c87-49f4-9eb4-c66586dd1fd2" (UID: "b436852d-3c87-49f4-9eb4-c66586dd1fd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:20.021704 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:20.021675    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b436852d-3c87-49f4-9eb4-c66586dd1fd2-maas-api-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:20.023606 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:20.023580    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b436852d-3c87-49f4-9eb4-c66586dd1fd2-kube-api-access-h5lk9" pod "b436852d-3c87-49f4-9eb4-c66586dd1fd2" (UID: "b436852d-3c87-49f4-9eb4-c66586dd1fd2"). InnerVolumeSpecName "kube-api-access-h5lk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:20.123276 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:20.123236    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5lk9\" (UniqueName: \"kubernetes.io/projected/b436852d-3c87-49f4-9eb4-c66586dd1fd2-kube-api-access-h5lk9\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:20.622250 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:20.622216    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a050ec-c2ba-4bf4-a625-36ce24f78a62" path="/var/lib/kubelet/pods/22a050ec-c2ba-4bf4-a625-36ce24f78a62/volumes"
Oct 31 04:00:20.622651 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:20.622638    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d03d8d7-4995-493f-93b5-0a601eccc765" path="/var/lib/kubelet/pods/8d03d8d7-4995-493f-93b5-0a601eccc765/volumes"
Oct 31 04:00:20.684993 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:20.684962    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"
Oct 31 04:00:20.684993 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:20.684977    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj" event={"ID":"b436852d-3c87-49f4-9eb4-c66586dd1fd2","Type":"ContainerDied","Data":"95e7a6b6edf8dcc5ecd192c6a04bacbe0eedabc22b8b36bb87dc38f146ad2e44"}
Oct 31 04:00:20.685238 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:20.685036    5415 scope.go:122] "RemoveContainer" containerID="25c91d0a584f411c7c979b7a656bac5437f2f6adac709cd2dc779d43c1b5ee28"
Oct 31 04:00:20.841893 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:20.841845    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"]
Oct 31 04:00:20.846072 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:20.846042    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890320-knslj"]
Oct 31 04:00:22.606240 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:00:22.606203    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b436852d-3c87-49f4-9eb4-c66586dd1fd2" path="/var/lib/kubelet/pods/b436852d-3c87-49f4-9eb4-c66586dd1fd2/volumes"
Oct 31 04:00:46.177809 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:00:46.177761    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:01:16.180421 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:01:16.179870    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:01:18.078700 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:18.078656    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"]
Oct 31 04:01:33.099211 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.099163    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls" podUID="78d5705e-0fee-477d-a5c1-c02375c63ea5" containerName="main" containerID="cri-o://bf7d474ccf79aba5623120e852993e96e6b06f48f5946f3fb9db5a8db542504e" gracePeriod=45
Oct 31 04:01:33.371674 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.371644    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 04:01:33.496635 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.496603    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-home\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-home\") pod \"78d5705e-0fee-477d-a5c1-c02375c63ea5\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") "
Oct 31 04:01:33.496838 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.496681    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-tmp-dir\") pod \"78d5705e-0fee-477d-a5c1-c02375c63ea5\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") "
Oct 31 04:01:33.496838 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.496710    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-dshm\") pod \"78d5705e-0fee-477d-a5c1-c02375c63ea5\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") "
Oct 31 04:01:33.496838 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.496738    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-model-cache\") pod \"78d5705e-0fee-477d-a5c1-c02375c63ea5\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") "
Oct 31 04:01:33.496838 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.496756    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78d5705e-0fee-477d-a5c1-c02375c63ea5-tls-certs\" (UniqueName: \"kubernetes.io/secret/78d5705e-0fee-477d-a5c1-c02375c63ea5-tls-certs\") pod \"78d5705e-0fee-477d-a5c1-c02375c63ea5\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") "
Oct 31 04:01:33.496838 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.496807    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d5705e-0fee-477d-a5c1-c02375c63ea5-kube-api-access-frs4r\" (UniqueName: \"kubernetes.io/projected/78d5705e-0fee-477d-a5c1-c02375c63ea5-kube-api-access-frs4r\") pod \"78d5705e-0fee-477d-a5c1-c02375c63ea5\" (UID: \"78d5705e-0fee-477d-a5c1-c02375c63ea5\") "
Oct 31 04:01:33.497130 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.496945    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-home" pod "78d5705e-0fee-477d-a5c1-c02375c63ea5" (UID: "78d5705e-0fee-477d-a5c1-c02375c63ea5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:33.497130 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.497043    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-tmp-dir" pod "78d5705e-0fee-477d-a5c1-c02375c63ea5" (UID: "78d5705e-0fee-477d-a5c1-c02375c63ea5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:33.497130 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.497059    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-model-cache" pod "78d5705e-0fee-477d-a5c1-c02375c63ea5" (UID: "78d5705e-0fee-477d-a5c1-c02375c63ea5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:33.497370 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.497347    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:33.497475 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.497374    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:33.497475 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.497388    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:33.499742 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.499719    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d5705e-0fee-477d-a5c1-c02375c63ea5-tls-certs" pod "78d5705e-0fee-477d-a5c1-c02375c63ea5" (UID: "78d5705e-0fee-477d-a5c1-c02375c63ea5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:01:33.499845 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.499800    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d5705e-0fee-477d-a5c1-c02375c63ea5-kube-api-access-frs4r" pod "78d5705e-0fee-477d-a5c1-c02375c63ea5" (UID: "78d5705e-0fee-477d-a5c1-c02375c63ea5"). InnerVolumeSpecName "kube-api-access-frs4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:33.499980 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.499962    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-dshm" pod "78d5705e-0fee-477d-a5c1-c02375c63ea5" (UID: "78d5705e-0fee-477d-a5c1-c02375c63ea5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:33.606960 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.599546    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78d5705e-0fee-477d-a5c1-c02375c63ea5-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:33.606960 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.599588    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78d5705e-0fee-477d-a5c1-c02375c63ea5-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:33.606960 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:33.599603    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frs4r\" (UniqueName: \"kubernetes.io/projected/78d5705e-0fee-477d-a5c1-c02375c63ea5-kube-api-access-frs4r\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:34.115117 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.115080    5415 generic.go:356] "Generic (PLEG): container finished" podID="78d5705e-0fee-477d-a5c1-c02375c63ea5" containerID="bf7d474ccf79aba5623120e852993e96e6b06f48f5946f3fb9db5a8db542504e" exitCode=0
Oct 31 04:01:34.115589 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.115148    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"
Oct 31 04:01:34.115589 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.115145    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls" event={"ID":"78d5705e-0fee-477d-a5c1-c02375c63ea5","Type":"ContainerDied","Data":"bf7d474ccf79aba5623120e852993e96e6b06f48f5946f3fb9db5a8db542504e"}
Oct 31 04:01:34.115589 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.115196    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls" event={"ID":"78d5705e-0fee-477d-a5c1-c02375c63ea5","Type":"ContainerDied","Data":"f0486543cb3df68ee82fbf53448df21f9acb3b65e63779aa302a9d6c708a31c2"}
Oct 31 04:01:34.115589 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.115218    5415 scope.go:122] "RemoveContainer" containerID="bf7d474ccf79aba5623120e852993e96e6b06f48f5946f3fb9db5a8db542504e"
Oct 31 04:01:34.126836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.126816    5415 scope.go:122] "RemoveContainer" containerID="bf7d474ccf79aba5623120e852993e96e6b06f48f5946f3fb9db5a8db542504e"
Oct 31 04:01:34.127200 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:01:34.127177    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7d474ccf79aba5623120e852993e96e6b06f48f5946f3fb9db5a8db542504e\": container with ID starting with bf7d474ccf79aba5623120e852993e96e6b06f48f5946f3fb9db5a8db542504e not found: ID does not exist" containerID="bf7d474ccf79aba5623120e852993e96e6b06f48f5946f3fb9db5a8db542504e"
Oct 31 04:01:34.127253 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.127211    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7d474ccf79aba5623120e852993e96e6b06f48f5946f3fb9db5a8db542504e"} err="failed to get container status \"bf7d474ccf79aba5623120e852993e96e6b06f48f5946f3fb9db5a8db542504e\": rpc error: code = NotFound desc = could not find container \"bf7d474ccf79aba5623120e852993e96e6b06f48f5946f3fb9db5a8db542504e\": container with ID starting with bf7d474ccf79aba5623120e852993e96e6b06f48f5946f3fb9db5a8db542504e not found: ID does not exist"
Oct 31 04:01:34.270016 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.269981    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"]
Oct 31 04:01:34.316694 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.316654    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2339ed/auth-test-model-2339ed-kserve-64788fb64f-trqls"]
Oct 31 04:01:34.416309 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.416212    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"]
Oct 31 04:01:34.606226 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.606190    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d5705e-0fee-477d-a5c1-c02375c63ea5" path="/var/lib/kubelet/pods/78d5705e-0fee-477d-a5c1-c02375c63ea5/volumes"
Oct 31 04:01:34.687594 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.687507    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"]
Oct 31 04:01:34.687781 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.687758    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" podUID="dc4cf674-d047-4b52-9081-1d89348f2546" containerName="istio-proxy" containerID="cri-o://4fd3cfee241df65d26b1d0f258c5fd3e58105132f0083372cfa20a36b96f8e0d" gracePeriod=30
Oct 31 04:01:34.690534 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.690508    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"]
Oct 31 04:01:34.690804 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:34.690772    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm" podUID="021b11b1-eadc-4435-9bcc-9bac04de2bf8" containerName="istio-proxy" containerID="cri-o://06ef4e51e9946f024fc7296e975d4a39ad28bdd48e94ddac6d66e210302dfce0" gracePeriod=30
Oct 31 04:01:38.181109 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:38.181074    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:01:38.181550 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:38.181132    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" podUID="dc4cf674-d047-4b52-9081-1d89348f2546" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:01:39.359863 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:39.359820    5415 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:01:39.360273 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:39.359915    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm" podUID="021b11b1-eadc-4435-9bcc-9bac04de2bf8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:01:39.417066 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:39.417024    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6" podUID="5e251604-6dee-4850-be98-ad59356e3038" containerName="payload-processing" containerID="cri-o://cd29c96b1995bda709c1a2639d98126d5b4aa69ca8201c5af15b98640423db9a" gracePeriod=25
Oct 31 04:01:39.985746 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:39.985718    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 04:01:40.006741 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.006713    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 04:01:40.070375 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.070341    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-kube-api-access-kld4n\" (UniqueName: \"kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-kube-api-access-kld4n\") pod \"dc4cf674-d047-4b52-9081-1d89348f2546\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") "
Oct 31 04:01:40.070562 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.070395    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-istio-token\" (UniqueName: \"kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-istio-token\") pod \"dc4cf674-d047-4b52-9081-1d89348f2546\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") "
Oct 31 04:01:40.070562 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.070437    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc4cf674-d047-4b52-9081-1d89348f2546-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4cf674-d047-4b52-9081-1d89348f2546-istio-podinfo\") pod \"dc4cf674-d047-4b52-9081-1d89348f2546\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") "
Oct 31 04:01:40.070562 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.070489    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-certs\") pod \"dc4cf674-d047-4b52-9081-1d89348f2546\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") "
Oct 31 04:01:40.070562 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.070525    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-socket\") pod \"dc4cf674-d047-4b52-9081-1d89348f2546\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") "
Oct 31 04:01:40.070562 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.070553    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4cf674-d047-4b52-9081-1d89348f2546-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4cf674-d047-4b52-9081-1d89348f2546-istiod-ca-cert\") pod \"dc4cf674-d047-4b52-9081-1d89348f2546\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") "
Oct 31 04:01:40.070850 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.070594    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-envoy\") pod \"dc4cf674-d047-4b52-9081-1d89348f2546\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") "
Oct 31 04:01:40.070850 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.070690    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-data\") pod \"dc4cf674-d047-4b52-9081-1d89348f2546\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") "
Oct 31 04:01:40.070850 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.070745    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-credential-socket\") pod \"dc4cf674-d047-4b52-9081-1d89348f2546\" (UID: \"dc4cf674-d047-4b52-9081-1d89348f2546\") "
Oct 31 04:01:40.070850 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.070809    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-certs" pod "dc4cf674-d047-4b52-9081-1d89348f2546" (UID: "dc4cf674-d047-4b52-9081-1d89348f2546"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:40.070850 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.070826    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-socket" pod "dc4cf674-d047-4b52-9081-1d89348f2546" (UID: "dc4cf674-d047-4b52-9081-1d89348f2546"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:40.071180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.071116    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-data" pod "dc4cf674-d047-4b52-9081-1d89348f2546" (UID: "dc4cf674-d047-4b52-9081-1d89348f2546"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:40.071239 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.071211    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-credential-socket" pod "dc4cf674-d047-4b52-9081-1d89348f2546" (UID: "dc4cf674-d047-4b52-9081-1d89348f2546"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:40.071298 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.071288    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.071355 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.071301    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.071355 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.071312    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.071480 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.071452    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4cf674-d047-4b52-9081-1d89348f2546-istiod-ca-cert" pod "dc4cf674-d047-4b52-9081-1d89348f2546" (UID: "dc4cf674-d047-4b52-9081-1d89348f2546"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:40.072745 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.072719    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc4cf674-d047-4b52-9081-1d89348f2546-istio-podinfo" pod "dc4cf674-d047-4b52-9081-1d89348f2546" (UID: "dc4cf674-d047-4b52-9081-1d89348f2546"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:01:40.073031 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.072998    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-envoy" pod "dc4cf674-d047-4b52-9081-1d89348f2546" (UID: "dc4cf674-d047-4b52-9081-1d89348f2546"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:40.073130 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.073114    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-istio-token" pod "dc4cf674-d047-4b52-9081-1d89348f2546" (UID: "dc4cf674-d047-4b52-9081-1d89348f2546"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:40.073180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.073139    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-kube-api-access-kld4n" pod "dc4cf674-d047-4b52-9081-1d89348f2546" (UID: "dc4cf674-d047-4b52-9081-1d89348f2546"). InnerVolumeSpecName "kube-api-access-kld4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:40.155467 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.155434    5415 generic.go:356] "Generic (PLEG): container finished" podID="021b11b1-eadc-4435-9bcc-9bac04de2bf8" containerID="06ef4e51e9946f024fc7296e975d4a39ad28bdd48e94ddac6d66e210302dfce0" exitCode=0
Oct 31 04:01:40.155654 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.155545    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"
Oct 31 04:01:40.155654 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.155570    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm" event={"ID":"021b11b1-eadc-4435-9bcc-9bac04de2bf8","Type":"ContainerDied","Data":"06ef4e51e9946f024fc7296e975d4a39ad28bdd48e94ddac6d66e210302dfce0"}
Oct 31 04:01:40.155779 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.155660    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm" event={"ID":"021b11b1-eadc-4435-9bcc-9bac04de2bf8","Type":"ContainerDied","Data":"9034737413dc87c84012caf0b19e49be0a3bac2ee5d6cd174314d198af42d1b2"}
Oct 31 04:01:40.155779 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.155691    5415 scope.go:122] "RemoveContainer" containerID="06ef4e51e9946f024fc7296e975d4a39ad28bdd48e94ddac6d66e210302dfce0"
Oct 31 04:01:40.157012 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.156991    5415 generic.go:356] "Generic (PLEG): container finished" podID="dc4cf674-d047-4b52-9081-1d89348f2546" containerID="4fd3cfee241df65d26b1d0f258c5fd3e58105132f0083372cfa20a36b96f8e0d" exitCode=0
Oct 31 04:01:40.157118 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.157041    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"
Oct 31 04:01:40.157118 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.157073    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" event={"ID":"dc4cf674-d047-4b52-9081-1d89348f2546","Type":"ContainerDied","Data":"4fd3cfee241df65d26b1d0f258c5fd3e58105132f0083372cfa20a36b96f8e0d"}
Oct 31 04:01:40.157216 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.157121    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8" event={"ID":"dc4cf674-d047-4b52-9081-1d89348f2546","Type":"ContainerDied","Data":"e5949cada8d713f1c2d69d9b57dd9002bea3c51c791029b6120e46b9809eae1a"}
Oct 31 04:01:40.167590 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.167566    5415 scope.go:122] "RemoveContainer" containerID="06ef4e51e9946f024fc7296e975d4a39ad28bdd48e94ddac6d66e210302dfce0"
Oct 31 04:01:40.168136 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:01:40.167995    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ef4e51e9946f024fc7296e975d4a39ad28bdd48e94ddac6d66e210302dfce0\": container with ID starting with 06ef4e51e9946f024fc7296e975d4a39ad28bdd48e94ddac6d66e210302dfce0 not found: ID does not exist" containerID="06ef4e51e9946f024fc7296e975d4a39ad28bdd48e94ddac6d66e210302dfce0"
Oct 31 04:01:40.168136 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.168030    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ef4e51e9946f024fc7296e975d4a39ad28bdd48e94ddac6d66e210302dfce0"} err="failed to get container status \"06ef4e51e9946f024fc7296e975d4a39ad28bdd48e94ddac6d66e210302dfce0\": rpc error: code = NotFound desc = could not find container \"06ef4e51e9946f024fc7296e975d4a39ad28bdd48e94ddac6d66e210302dfce0\": container with ID starting with 06ef4e51e9946f024fc7296e975d4a39ad28bdd48e94ddac6d66e210302dfce0 not found: ID does not exist"
Oct 31 04:01:40.168136 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.168053    5415 scope.go:122] "RemoveContainer" containerID="4fd3cfee241df65d26b1d0f258c5fd3e58105132f0083372cfa20a36b96f8e0d"
Oct 31 04:01:40.171928 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.171902    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-kube-api-access-mj5gl\" (UniqueName: \"kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-kube-api-access-mj5gl\") pod \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") "
Oct 31 04:01:40.172041 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.171963    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-data\") pod \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") "
Oct 31 04:01:40.172041 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.172000    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istiod-ca-cert\") pod \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") "
Oct 31 04:01:40.172041 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.172034    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-envoy\") pod \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") "
Oct 31 04:01:40.172210 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.172110    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-socket\") pod \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") "
Oct 31 04:01:40.172210 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.172202    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-token\" (UniqueName: \"kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-token\") pod \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") "
Oct 31 04:01:40.172326 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.172262    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-certs\") pod \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") "
Oct 31 04:01:40.172558 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.172527    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-socket" pod "021b11b1-eadc-4435-9bcc-9bac04de2bf8" (UID: "021b11b1-eadc-4435-9bcc-9bac04de2bf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:40.172558 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.172545    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-certs" pod "021b11b1-eadc-4435-9bcc-9bac04de2bf8" (UID: "021b11b1-eadc-4435-9bcc-9bac04de2bf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:40.172699 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.172593    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-credential-socket\") pod \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") "
Oct 31 04:01:40.172699 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.172606    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istiod-ca-cert" pod "021b11b1-eadc-4435-9bcc-9bac04de2bf8" (UID: "021b11b1-eadc-4435-9bcc-9bac04de2bf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:40.172806 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.172696    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-podinfo\") pod \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\" (UID: \"021b11b1-eadc-4435-9bcc-9bac04de2bf8\") "
Oct 31 04:01:40.172856 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.172841    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-data" pod "021b11b1-eadc-4435-9bcc-9bac04de2bf8" (UID: "021b11b1-eadc-4435-9bcc-9bac04de2bf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:40.172960 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.172857    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-credential-socket" pod "021b11b1-eadc-4435-9bcc-9bac04de2bf8" (UID: "021b11b1-eadc-4435-9bcc-9bac04de2bf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:40.173395 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.173235    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kld4n\" (UniqueName: \"kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-kube-api-access-kld4n\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.173395 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.173262    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4cf674-d047-4b52-9081-1d89348f2546-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.173395 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.173278    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4cf674-d047-4b52-9081-1d89348f2546-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.173395 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.173292    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.173395 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.173301    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.173395 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.173310    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4cf674-d047-4b52-9081-1d89348f2546-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.173395 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.173319    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.173395 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.173329    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.173395 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.173342    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4cf674-d047-4b52-9081-1d89348f2546-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.173395 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.173355    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.173395 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.173372    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.181611 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.181589    5415 scope.go:122] "RemoveContainer" containerID="4fd3cfee241df65d26b1d0f258c5fd3e58105132f0083372cfa20a36b96f8e0d"
Oct 31 04:01:40.181938 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:01:40.181913    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd3cfee241df65d26b1d0f258c5fd3e58105132f0083372cfa20a36b96f8e0d\": container with ID starting with 4fd3cfee241df65d26b1d0f258c5fd3e58105132f0083372cfa20a36b96f8e0d not found: ID does not exist" containerID="4fd3cfee241df65d26b1d0f258c5fd3e58105132f0083372cfa20a36b96f8e0d"
Oct 31 04:01:40.182027 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.181945    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd3cfee241df65d26b1d0f258c5fd3e58105132f0083372cfa20a36b96f8e0d"} err="failed to get container status \"4fd3cfee241df65d26b1d0f258c5fd3e58105132f0083372cfa20a36b96f8e0d\": rpc error: code = NotFound desc = could not find container \"4fd3cfee241df65d26b1d0f258c5fd3e58105132f0083372cfa20a36b96f8e0d\": container with ID starting with 4fd3cfee241df65d26b1d0f258c5fd3e58105132f0083372cfa20a36b96f8e0d not found: ID does not exist"
Oct 31 04:01:40.183240 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.183220    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-kube-api-access-mj5gl" pod "021b11b1-eadc-4435-9bcc-9bac04de2bf8" (UID: "021b11b1-eadc-4435-9bcc-9bac04de2bf8"). InnerVolumeSpecName "kube-api-access-mj5gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:40.193364 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.193333    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-envoy" pod "021b11b1-eadc-4435-9bcc-9bac04de2bf8" (UID: "021b11b1-eadc-4435-9bcc-9bac04de2bf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:40.193364 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.193350    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-token" pod "021b11b1-eadc-4435-9bcc-9bac04de2bf8" (UID: "021b11b1-eadc-4435-9bcc-9bac04de2bf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:40.193364 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.193342    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-podinfo" pod "021b11b1-eadc-4435-9bcc-9bac04de2bf8" (UID: "021b11b1-eadc-4435-9bcc-9bac04de2bf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:01:40.274635 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.274550    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.274635 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.274591    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.274635 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.274601    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021b11b1-eadc-4435-9bcc-9bac04de2bf8-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.274635 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.274611    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj5gl\" (UniqueName: \"kubernetes.io/projected/021b11b1-eadc-4435-9bcc-9bac04de2bf8-kube-api-access-mj5gl\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.411141 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.411100    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"]
Oct 31 04:01:40.414134 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.414106    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-gq2j8"]
Oct 31 04:01:40.570742 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.570646    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"]
Oct 31 04:01:40.574483 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.574454    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-cf5lm"]
Oct 31 04:01:40.606016 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.605984    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="021b11b1-eadc-4435-9bcc-9bac04de2bf8" path="/var/lib/kubelet/pods/021b11b1-eadc-4435-9bcc-9bac04de2bf8/volumes"
Oct 31 04:01:40.606436 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:40.606423    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4cf674-d047-4b52-9081-1d89348f2546" path="/var/lib/kubelet/pods/dc4cf674-d047-4b52-9081-1d89348f2546/volumes"
Oct 31 04:01:44.706272 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:44.706232    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6" podUID="5e251604-6dee-4850-be98-ad59356e3038" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:01:46.175918 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:01:46.175806    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:01:54.706104 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:54.706064    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6" podUID="5e251604-6dee-4850-be98-ad59356e3038" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:01:59.689091 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:59.689022    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"
Oct 31 04:01:59.795448 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:59.795418    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e251604-6dee-4850-be98-ad59356e3038-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e251604-6dee-4850-be98-ad59356e3038-plugins-config-volume\") pod \"5e251604-6dee-4850-be98-ad59356e3038\" (UID: \"5e251604-6dee-4850-be98-ad59356e3038\") "
Oct 31 04:01:59.795641 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:59.795507    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e251604-6dee-4850-be98-ad59356e3038-kube-api-access-kbklw\" (UniqueName: \"kubernetes.io/projected/5e251604-6dee-4850-be98-ad59356e3038-kube-api-access-kbklw\") pod \"5e251604-6dee-4850-be98-ad59356e3038\" (UID: \"5e251604-6dee-4850-be98-ad59356e3038\") "
Oct 31 04:01:59.795808 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:59.795784    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e251604-6dee-4850-be98-ad59356e3038-plugins-config-volume" pod "5e251604-6dee-4850-be98-ad59356e3038" (UID: "5e251604-6dee-4850-be98-ad59356e3038"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:59.795933 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:59.795908    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e251604-6dee-4850-be98-ad59356e3038-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:59.797749 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:59.797714    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e251604-6dee-4850-be98-ad59356e3038-kube-api-access-kbklw" pod "5e251604-6dee-4850-be98-ad59356e3038" (UID: "5e251604-6dee-4850-be98-ad59356e3038"). InnerVolumeSpecName "kube-api-access-kbklw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:59.897406 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:01:59.897366    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbklw\" (UniqueName: \"kubernetes.io/projected/5e251604-6dee-4850-be98-ad59356e3038-kube-api-access-kbklw\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:00.275576 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:00.275448    5415 generic.go:356] "Generic (PLEG): container finished" podID="5e251604-6dee-4850-be98-ad59356e3038" containerID="cd29c96b1995bda709c1a2639d98126d5b4aa69ca8201c5af15b98640423db9a" exitCode=0
Oct 31 04:02:00.275576 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:00.275491    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"
Oct 31 04:02:00.275576 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:00.275521    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6" event={"ID":"5e251604-6dee-4850-be98-ad59356e3038","Type":"ContainerDied","Data":"cd29c96b1995bda709c1a2639d98126d5b4aa69ca8201c5af15b98640423db9a"}
Oct 31 04:02:00.275576 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:00.275569    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6" event={"ID":"5e251604-6dee-4850-be98-ad59356e3038","Type":"ContainerDied","Data":"3204169aace17159db20df69df3562a912be82750cbec415da537546a8f06056"}
Oct 31 04:02:00.305226 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:00.275590    5415 scope.go:122] "RemoveContainer" containerID="cd29c96b1995bda709c1a2639d98126d5b4aa69ca8201c5af15b98640423db9a"
Oct 31 04:02:00.305226 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:00.285589    5415 scope.go:122] "RemoveContainer" containerID="cd29c96b1995bda709c1a2639d98126d5b4aa69ca8201c5af15b98640423db9a"
Oct 31 04:02:00.305226 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:02:00.285921    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd29c96b1995bda709c1a2639d98126d5b4aa69ca8201c5af15b98640423db9a\": container with ID starting with cd29c96b1995bda709c1a2639d98126d5b4aa69ca8201c5af15b98640423db9a not found: ID does not exist" containerID="cd29c96b1995bda709c1a2639d98126d5b4aa69ca8201c5af15b98640423db9a"
Oct 31 04:02:00.305226 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:00.285951    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd29c96b1995bda709c1a2639d98126d5b4aa69ca8201c5af15b98640423db9a"} err="failed to get container status \"cd29c96b1995bda709c1a2639d98126d5b4aa69ca8201c5af15b98640423db9a\": rpc error: code = NotFound desc = could not find container \"cd29c96b1995bda709c1a2639d98126d5b4aa69ca8201c5af15b98640423db9a\": container with ID starting with cd29c96b1995bda709c1a2639d98126d5b4aa69ca8201c5af15b98640423db9a not found: ID does not exist"
Oct 31 04:02:00.429847 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:00.429755    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"]
Oct 31 04:02:00.476787 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:00.476751    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ab9b62-6cdd9fbb87-xsbm6"]
Oct 31 04:02:00.605299 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:00.605269    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e251604-6dee-4850-be98-ad59356e3038" path="/var/lib/kubelet/pods/5e251604-6dee-4850-be98-ad59356e3038/volumes"
Oct 31 04:02:16.176091 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:02:16.176055    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:02:46.175897 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:02:46.175837    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:02:51.733890 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:51.733838    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"]
Oct 31 04:02:51.734486 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:51.734067    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b" podUID="7ad16f86-742e-4a4a-8e24-9de588505c03" containerName="maas-api" containerID="cri-o://dfb267bc7bb7ea17cea5ea007facddb8b7f8d801ffed8580abc5fd5c8239456d" gracePeriod=30
Oct 31 04:02:52.021481 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.021444    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 04:02:52.083129 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.083094    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad16f86-742e-4a4a-8e24-9de588505c03-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad16f86-742e-4a4a-8e24-9de588505c03-openshift-service-ca\") pod \"7ad16f86-742e-4a4a-8e24-9de588505c03\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") "
Oct 31 04:02:52.083344 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.083217    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad16f86-742e-4a4a-8e24-9de588505c03-kube-api-access-7sgkw\" (UniqueName: \"kubernetes.io/projected/7ad16f86-742e-4a4a-8e24-9de588505c03-kube-api-access-7sgkw\") pod \"7ad16f86-742e-4a4a-8e24-9de588505c03\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") "
Oct 31 04:02:52.083344 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.083244    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-metrics-certs\") pod \"7ad16f86-742e-4a4a-8e24-9de588505c03\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") "
Oct 31 04:02:52.083462 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.083341    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-maas-api-tls\") pod \"7ad16f86-742e-4a4a-8e24-9de588505c03\" (UID: \"7ad16f86-742e-4a4a-8e24-9de588505c03\") "
Oct 31 04:02:52.083617 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.083571    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad16f86-742e-4a4a-8e24-9de588505c03-openshift-service-ca" pod "7ad16f86-742e-4a4a-8e24-9de588505c03" (UID: "7ad16f86-742e-4a4a-8e24-9de588505c03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:02:52.083817 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.083790    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad16f86-742e-4a4a-8e24-9de588505c03-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:52.085612 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.085564    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-maas-api-tls" pod "7ad16f86-742e-4a4a-8e24-9de588505c03" (UID: "7ad16f86-742e-4a4a-8e24-9de588505c03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:02:52.085736 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.085667    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-metrics-certs" pod "7ad16f86-742e-4a4a-8e24-9de588505c03" (UID: "7ad16f86-742e-4a4a-8e24-9de588505c03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:02:52.085988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.085965    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad16f86-742e-4a4a-8e24-9de588505c03-kube-api-access-7sgkw" pod "7ad16f86-742e-4a4a-8e24-9de588505c03" (UID: "7ad16f86-742e-4a4a-8e24-9de588505c03"). InnerVolumeSpecName "kube-api-access-7sgkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:52.185512 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.185455    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:52.185512 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.185497    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad16f86-742e-4a4a-8e24-9de588505c03-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:52.185512 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.185513    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sgkw\" (UniqueName: \"kubernetes.io/projected/7ad16f86-742e-4a4a-8e24-9de588505c03-kube-api-access-7sgkw\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:52.589385 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.589347    5415 generic.go:356] "Generic (PLEG): container finished" podID="7ad16f86-742e-4a4a-8e24-9de588505c03" containerID="dfb267bc7bb7ea17cea5ea007facddb8b7f8d801ffed8580abc5fd5c8239456d" exitCode=0
Oct 31 04:02:52.589585 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.589398    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"
Oct 31 04:02:52.589585 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.589446    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b" event={"ID":"7ad16f86-742e-4a4a-8e24-9de588505c03","Type":"ContainerDied","Data":"dfb267bc7bb7ea17cea5ea007facddb8b7f8d801ffed8580abc5fd5c8239456d"}
Oct 31 04:02:52.589585 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.589503    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b" event={"ID":"7ad16f86-742e-4a4a-8e24-9de588505c03","Type":"ContainerDied","Data":"b5b1b1950c7e4dcc3321339aed23c131a6d6a442d61043ce62314e8aa08a6126"}
Oct 31 04:02:52.589585 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.589525    5415 scope.go:122] "RemoveContainer" containerID="dfb267bc7bb7ea17cea5ea007facddb8b7f8d801ffed8580abc5fd5c8239456d"
Oct 31 04:02:52.600091 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.600070    5415 scope.go:122] "RemoveContainer" containerID="dfb267bc7bb7ea17cea5ea007facddb8b7f8d801ffed8580abc5fd5c8239456d"
Oct 31 04:02:52.600451 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:02:52.600432    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb267bc7bb7ea17cea5ea007facddb8b7f8d801ffed8580abc5fd5c8239456d\": container with ID starting with dfb267bc7bb7ea17cea5ea007facddb8b7f8d801ffed8580abc5fd5c8239456d not found: ID does not exist" containerID="dfb267bc7bb7ea17cea5ea007facddb8b7f8d801ffed8580abc5fd5c8239456d"
Oct 31 04:02:52.600524 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.600460    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb267bc7bb7ea17cea5ea007facddb8b7f8d801ffed8580abc5fd5c8239456d"} err="failed to get container status \"dfb267bc7bb7ea17cea5ea007facddb8b7f8d801ffed8580abc5fd5c8239456d\": rpc error: code = NotFound desc = could not find container \"dfb267bc7bb7ea17cea5ea007facddb8b7f8d801ffed8580abc5fd5c8239456d\": container with ID starting with dfb267bc7bb7ea17cea5ea007facddb8b7f8d801ffed8580abc5fd5c8239456d not found: ID does not exist"
Oct 31 04:02:52.743985 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.743947    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"]
Oct 31 04:02:52.790699 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:52.790655    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9b62-59789cd6f6-c6h4b"]
Oct 31 04:02:54.606026 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:54.605989    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad16f86-742e-4a4a-8e24-9de588505c03" path="/var/lib/kubelet/pods/7ad16f86-742e-4a4a-8e24-9de588505c03/volumes"
Oct 31 04:02:58.596369 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596333    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"]
Oct 31 04:02:58.596855 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596840    5415 cpu_manager.go:414] "containerMap: removing container" podUID="78d5705e-0fee-477d-a5c1-c02375c63ea5" containerName="main"
Oct 31 04:02:58.596933 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596856    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d5705e-0fee-477d-a5c1-c02375c63ea5" containerName="main"
Oct 31 04:02:58.596933 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596870    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8d03d8d7-4995-493f-93b5-0a601eccc765" containerName="cleanup"
Oct 31 04:02:58.596933 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596893    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d03d8d7-4995-493f-93b5-0a601eccc765" containerName="cleanup"
Oct 31 04:02:58.596933 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596909    5415 cpu_manager.go:414] "containerMap: removing container" podUID="22a050ec-c2ba-4bf4-a625-36ce24f78a62" containerName="cleanup"
Oct 31 04:02:58.596933 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596915    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a050ec-c2ba-4bf4-a625-36ce24f78a62" containerName="cleanup"
Oct 31 04:02:58.596933 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596923    5415 cpu_manager.go:414] "containerMap: removing container" podUID="5e251604-6dee-4850-be98-ad59356e3038" containerName="payload-processing"
Oct 31 04:02:58.596933 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596929    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e251604-6dee-4850-be98-ad59356e3038" containerName="payload-processing"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596940    5415 cpu_manager.go:414] "containerMap: removing container" podUID="7ad16f86-742e-4a4a-8e24-9de588505c03" containerName="maas-api"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596945    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad16f86-742e-4a4a-8e24-9de588505c03" containerName="maas-api"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596953    5415 cpu_manager.go:414] "containerMap: removing container" podUID="22a050ec-c2ba-4bf4-a625-36ce24f78a62" containerName="cleanup"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596957    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a050ec-c2ba-4bf4-a625-36ce24f78a62" containerName="cleanup"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596963    5415 cpu_manager.go:414] "containerMap: removing container" podUID="b436852d-3c87-49f4-9eb4-c66586dd1fd2" containerName="cleanup"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596967    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b436852d-3c87-49f4-9eb4-c66586dd1fd2" containerName="cleanup"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596981    5415 cpu_manager.go:414] "containerMap: removing container" podUID="b436852d-3c87-49f4-9eb4-c66586dd1fd2" containerName="cleanup"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596985    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b436852d-3c87-49f4-9eb4-c66586dd1fd2" containerName="cleanup"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596993    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8d03d8d7-4995-493f-93b5-0a601eccc765" containerName="cleanup"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.596998    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d03d8d7-4995-493f-93b5-0a601eccc765" containerName="cleanup"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597011    5415 cpu_manager.go:414] "containerMap: removing container" podUID="021b11b1-eadc-4435-9bcc-9bac04de2bf8" containerName="istio-proxy"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597015    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021b11b1-eadc-4435-9bcc-9bac04de2bf8" containerName="istio-proxy"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597025    5415 cpu_manager.go:414] "containerMap: removing container" podUID="dc4cf674-d047-4b52-9081-1d89348f2546" containerName="istio-proxy"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597030    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4cf674-d047-4b52-9081-1d89348f2546" containerName="istio-proxy"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597106    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="021b11b1-eadc-4435-9bcc-9bac04de2bf8" containerName="istio-proxy"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597116    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d03d8d7-4995-493f-93b5-0a601eccc765" containerName="cleanup"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597122    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="b436852d-3c87-49f4-9eb4-c66586dd1fd2" containerName="cleanup"
Oct 31 04:02:58.597151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597128    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4cf674-d047-4b52-9081-1d89348f2546" containerName="istio-proxy"
Oct 31 04:02:58.597717 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597181    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d5705e-0fee-477d-a5c1-c02375c63ea5" containerName="main"
Oct 31 04:02:58.597717 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597193    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e251604-6dee-4850-be98-ad59356e3038" containerName="payload-processing"
Oct 31 04:02:58.597717 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597199    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d03d8d7-4995-493f-93b5-0a601eccc765" containerName="cleanup"
Oct 31 04:02:58.597717 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597204    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a050ec-c2ba-4bf4-a625-36ce24f78a62" containerName="cleanup"
Oct 31 04:02:58.597717 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597209    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad16f86-742e-4a4a-8e24-9de588505c03" containerName="maas-api"
Oct 31 04:02:58.597717 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597214    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a050ec-c2ba-4bf4-a625-36ce24f78a62" containerName="cleanup"
Oct 31 04:02:58.597717 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597229    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="b436852d-3c87-49f4-9eb4-c66586dd1fd2" containerName="cleanup"
Oct 31 04:02:58.597717 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.597275    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="b436852d-3c87-49f4-9eb4-c66586dd1fd2" containerName="cleanup"
Oct 31 04:02:58.601428 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.601390    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:58.650741 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.650703    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c7585c-serving-cert\""
Oct 31 04:02:58.660411 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.660361    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:58.660633 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.660452    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-metrics-certs\") pod \"maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:58.660633 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.660526    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl44d\" (UniqueName: \"kubernetes.io/projected/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-kube-api-access-fl44d\") pod \"maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:58.660767 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.660745    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:58.699928 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.699853    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"]
Oct 31 04:02:58.762131 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.762097    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:58.762346 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.762153    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:58.762346 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.762211    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-metrics-certs\") pod \"maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:58.762346 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.762240    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl44d\" (UniqueName: \"kubernetes.io/projected/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-kube-api-access-fl44d\") pod \"maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:58.762346 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:02:58.762273    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-serving-cert: secret "maas-api-e2e-extmod-a-c7585c-serving-cert" not found
Oct 31 04:02:58.762583 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:02:58.762364    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-maas-api-tls podName:dccd2ac0-5607-4a92-8a94-d32108a1b0b5 nodeName:}" failed. No retries permitted until 2026-10-31 04:02:59.262342941 +0000 UTC m=+3320.822822656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-maas-api-tls") pod "maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk" (UID: "dccd2ac0-5607-4a92-8a94-d32108a1b0b5") : secret "maas-api-e2e-extmod-a-c7585c-serving-cert" not found
Oct 31 04:02:58.762986 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.762963    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:58.764893 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.764849    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-metrics-certs\") pod \"maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:58.831779 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:58.831747    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl44d\" (UniqueName: \"kubernetes.io/projected/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-kube-api-access-fl44d\") pod \"maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:59.267294 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:59.267256    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:59.269760 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:59.269730    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:59.516131 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:59.516087    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:02:59.892622 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:59.892598    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"]
Oct 31 04:02:59.896056 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:02:59.896019    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccd2ac0_5607_4a92_8a94_d32108a1b0b5.slice/crio-0c3331c0f8c98cf74e4f423187a387ad404de067f48ec04a03702c214b88a2e2 WatchSource:0}: Error finding container 0c3331c0f8c98cf74e4f423187a387ad404de067f48ec04a03702c214b88a2e2: Status 404 returned error can't find the container with id 0c3331c0f8c98cf74e4f423187a387ad404de067f48ec04a03702c214b88a2e2
Oct 31 04:02:59.897639 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:02:59.897616    5415 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:03:00.638718 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:00.638675    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk" event={"ID":"dccd2ac0-5607-4a92-8a94-d32108a1b0b5","Type":"ContainerStarted","Data":"0c3331c0f8c98cf74e4f423187a387ad404de067f48ec04a03702c214b88a2e2"}
Oct 31 04:03:00.784832 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:00.784788    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"]
Oct 31 04:03:00.893591 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:00.893353    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"]
Oct 31 04:03:01.645680 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:01.645647    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk" event={"ID":"dccd2ac0-5607-4a92-8a94-d32108a1b0b5","Type":"ContainerStarted","Data":"b10710ef38cdc7e58a99d5803b6e0d0143eaf95ff3543c5c5b96408f5a7252fc"}
Oct 31 04:03:01.645892 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:01.645773    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:03:01.757089 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:01.757029    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk" podStartSLOduration=2.754416617 podStartE2EDuration="3.757013528s" totalImagesPullingTime="1.002596911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:01.755530325 +0000 UTC m=+3323.316010056" watchObservedRunningTime="2026-10-31 04:03:01.757013528 +0000 UTC m=+3323.317493343"
Oct 31 04:03:05.786056 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:05.785998    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z" podUID="8bbac2bb-21dd-483b-9cbc-d25ddcb6282c" containerName="payload-processing" containerID="cri-o://7deefeb825c8d72574f77b6829d3975a5807cb947e5da39672921cbad286b6fe" gracePeriod=25
Oct 31 04:03:05.893935 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:05.893884    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294" podUID="910e9857-1a61-41b9-a228-b39423e20d14" containerName="payload-pre-processing" containerID="cri-o://b30df98d7ec9a4f209d5fcd37b98a3d47c9436f365c17cb1e2be5437d1d41338" gracePeriod=25
Oct 31 04:03:06.058060 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.058025    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"
Oct 31 04:03:06.142281 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.142252    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-kube-api-access-4ftmz\" (UniqueName: \"kubernetes.io/projected/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-kube-api-access-4ftmz\") pod \"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c\" (UID: \"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c\") "
Oct 31 04:03:06.142442 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.142337    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-plugins-config-volume\") pod \"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c\" (UID: \"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c\") "
Oct 31 04:03:06.142955 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.142931    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-plugins-config-volume" pod "8bbac2bb-21dd-483b-9cbc-d25ddcb6282c" (UID: "8bbac2bb-21dd-483b-9cbc-d25ddcb6282c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:03:06.144648 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.144617    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-kube-api-access-4ftmz" pod "8bbac2bb-21dd-483b-9cbc-d25ddcb6282c" (UID: "8bbac2bb-21dd-483b-9cbc-d25ddcb6282c"). InnerVolumeSpecName "kube-api-access-4ftmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:03:06.152688 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.152658    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"
Oct 31 04:03:06.243656 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.243616    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/910e9857-1a61-41b9-a228-b39423e20d14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910e9857-1a61-41b9-a228-b39423e20d14-plugins-config-volume\") pod \"910e9857-1a61-41b9-a228-b39423e20d14\" (UID: \"910e9857-1a61-41b9-a228-b39423e20d14\") "
Oct 31 04:03:06.243848 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.243769    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/910e9857-1a61-41b9-a228-b39423e20d14-kube-api-access-v82l2\" (UniqueName: \"kubernetes.io/projected/910e9857-1a61-41b9-a228-b39423e20d14-kube-api-access-v82l2\") pod \"910e9857-1a61-41b9-a228-b39423e20d14\" (UID: \"910e9857-1a61-41b9-a228-b39423e20d14\") "
Oct 31 04:03:06.244053 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.244029    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/910e9857-1a61-41b9-a228-b39423e20d14-plugins-config-volume" pod "910e9857-1a61-41b9-a228-b39423e20d14" (UID: "910e9857-1a61-41b9-a228-b39423e20d14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:03:06.244238 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.244217    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ftmz\" (UniqueName: \"kubernetes.io/projected/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-kube-api-access-4ftmz\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:06.244238 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.244236    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910e9857-1a61-41b9-a228-b39423e20d14-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:06.244402 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.244246    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:06.245946 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.245914    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/910e9857-1a61-41b9-a228-b39423e20d14-kube-api-access-v82l2" pod "910e9857-1a61-41b9-a228-b39423e20d14" (UID: "910e9857-1a61-41b9-a228-b39423e20d14"). InnerVolumeSpecName "kube-api-access-v82l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:03:06.345963 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.345926    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v82l2\" (UniqueName: \"kubernetes.io/projected/910e9857-1a61-41b9-a228-b39423e20d14-kube-api-access-v82l2\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:06.675296 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.675257    5415 generic.go:356] "Generic (PLEG): container finished" podID="8bbac2bb-21dd-483b-9cbc-d25ddcb6282c" containerID="7deefeb825c8d72574f77b6829d3975a5807cb947e5da39672921cbad286b6fe" exitCode=0
Oct 31 04:03:06.675296 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.675301    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"
Oct 31 04:03:06.675582 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.675348    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z" event={"ID":"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c","Type":"ContainerDied","Data":"7deefeb825c8d72574f77b6829d3975a5807cb947e5da39672921cbad286b6fe"}
Oct 31 04:03:06.675582 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.675409    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z" event={"ID":"8bbac2bb-21dd-483b-9cbc-d25ddcb6282c","Type":"ContainerDied","Data":"e1272a1278f0d18f5dffac81286e953c236b11b26d3fb5bab04957f16a34c640"}
Oct 31 04:03:06.675582 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.675439    5415 scope.go:122] "RemoveContainer" containerID="7deefeb825c8d72574f77b6829d3975a5807cb947e5da39672921cbad286b6fe"
Oct 31 04:03:06.677149 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.677119    5415 generic.go:356] "Generic (PLEG): container finished" podID="910e9857-1a61-41b9-a228-b39423e20d14" containerID="b30df98d7ec9a4f209d5fcd37b98a3d47c9436f365c17cb1e2be5437d1d41338" exitCode=0
Oct 31 04:03:06.677293 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.677164    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294" event={"ID":"910e9857-1a61-41b9-a228-b39423e20d14","Type":"ContainerDied","Data":"b30df98d7ec9a4f209d5fcd37b98a3d47c9436f365c17cb1e2be5437d1d41338"}
Oct 31 04:03:06.677293 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.677171    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"
Oct 31 04:03:06.677293 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.677208    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294" event={"ID":"910e9857-1a61-41b9-a228-b39423e20d14","Type":"ContainerDied","Data":"e3f3c14bf93df58b4647523e889e70fa3e11cf03c19d031de0e08f7a65154cd4"}
Oct 31 04:03:06.685303 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.685280    5415 scope.go:122] "RemoveContainer" containerID="7deefeb825c8d72574f77b6829d3975a5807cb947e5da39672921cbad286b6fe"
Oct 31 04:03:06.685674 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:03:06.685642    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7deefeb825c8d72574f77b6829d3975a5807cb947e5da39672921cbad286b6fe\": container with ID starting with 7deefeb825c8d72574f77b6829d3975a5807cb947e5da39672921cbad286b6fe not found: ID does not exist" containerID="7deefeb825c8d72574f77b6829d3975a5807cb947e5da39672921cbad286b6fe"
Oct 31 04:03:06.685725 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.685678    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7deefeb825c8d72574f77b6829d3975a5807cb947e5da39672921cbad286b6fe"} err="failed to get container status \"7deefeb825c8d72574f77b6829d3975a5807cb947e5da39672921cbad286b6fe\": rpc error: code = NotFound desc = could not find container \"7deefeb825c8d72574f77b6829d3975a5807cb947e5da39672921cbad286b6fe\": container with ID starting with 7deefeb825c8d72574f77b6829d3975a5807cb947e5da39672921cbad286b6fe not found: ID does not exist"
Oct 31 04:03:06.685725 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.685703    5415 scope.go:122] "RemoveContainer" containerID="b30df98d7ec9a4f209d5fcd37b98a3d47c9436f365c17cb1e2be5437d1d41338"
Oct 31 04:03:06.695306 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.695278    5415 scope.go:122] "RemoveContainer" containerID="b30df98d7ec9a4f209d5fcd37b98a3d47c9436f365c17cb1e2be5437d1d41338"
Oct 31 04:03:06.695697 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:03:06.695669    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30df98d7ec9a4f209d5fcd37b98a3d47c9436f365c17cb1e2be5437d1d41338\": container with ID starting with b30df98d7ec9a4f209d5fcd37b98a3d47c9436f365c17cb1e2be5437d1d41338 not found: ID does not exist" containerID="b30df98d7ec9a4f209d5fcd37b98a3d47c9436f365c17cb1e2be5437d1d41338"
Oct 31 04:03:06.695793 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.695711    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30df98d7ec9a4f209d5fcd37b98a3d47c9436f365c17cb1e2be5437d1d41338"} err="failed to get container status \"b30df98d7ec9a4f209d5fcd37b98a3d47c9436f365c17cb1e2be5437d1d41338\": rpc error: code = NotFound desc = could not find container \"b30df98d7ec9a4f209d5fcd37b98a3d47c9436f365c17cb1e2be5437d1d41338\": container with ID starting with b30df98d7ec9a4f209d5fcd37b98a3d47c9436f365c17cb1e2be5437d1d41338 not found: ID does not exist"
Oct 31 04:03:06.830944 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.830904    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"]
Oct 31 04:03:06.834457 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.834424    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8b85a6-64f64c75c5-jk294"]
Oct 31 04:03:06.937608 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.937517    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"]
Oct 31 04:03:06.939517 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:06.939487    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8b85a6-fc886dbf-wv89z"]
Oct 31 04:03:07.656166 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:07.656132    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:03:08.606246 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:08.606207    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bbac2bb-21dd-483b-9cbc-d25ddcb6282c" path="/var/lib/kubelet/pods/8bbac2bb-21dd-483b-9cbc-d25ddcb6282c/volumes"
Oct 31 04:03:08.606676 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:08.606621    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="910e9857-1a61-41b9-a228-b39423e20d14" path="/var/lib/kubelet/pods/910e9857-1a61-41b9-a228-b39423e20d14/volumes"
Oct 31 04:03:16.177979 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:03:16.177871    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:03:31.003188 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:31.003149    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"]
Oct 31 04:03:43.957719 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.957685    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"]
Oct 31 04:03:43.958263 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958247    5415 cpu_manager.go:414] "containerMap: removing container" podUID="22a050ec-c2ba-4bf4-a625-36ce24f78a62" containerName="cleanup"
Oct 31 04:03:43.958307 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958264    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a050ec-c2ba-4bf4-a625-36ce24f78a62" containerName="cleanup"
Oct 31 04:03:43.958307 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958284    5415 cpu_manager.go:414] "containerMap: removing container" podUID="910e9857-1a61-41b9-a228-b39423e20d14" containerName="payload-pre-processing"
Oct 31 04:03:43.958307 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958292    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="910e9857-1a61-41b9-a228-b39423e20d14" containerName="payload-pre-processing"
Oct 31 04:03:43.958307 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958305    5415 cpu_manager.go:414] "containerMap: removing container" podUID="b436852d-3c87-49f4-9eb4-c66586dd1fd2" containerName="cleanup"
Oct 31 04:03:43.958433 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958310    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b436852d-3c87-49f4-9eb4-c66586dd1fd2" containerName="cleanup"
Oct 31 04:03:43.958433 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958324    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8bbac2bb-21dd-483b-9cbc-d25ddcb6282c" containerName="payload-processing"
Oct 31 04:03:43.958433 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958329    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bbac2bb-21dd-483b-9cbc-d25ddcb6282c" containerName="payload-processing"
Oct 31 04:03:43.958433 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958346    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8d03d8d7-4995-493f-93b5-0a601eccc765" containerName="cleanup"
Oct 31 04:03:43.958433 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958351    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d03d8d7-4995-493f-93b5-0a601eccc765" containerName="cleanup"
Oct 31 04:03:43.958577 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958445    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bbac2bb-21dd-483b-9cbc-d25ddcb6282c" containerName="payload-processing"
Oct 31 04:03:43.958577 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958454    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d03d8d7-4995-493f-93b5-0a601eccc765" containerName="cleanup"
Oct 31 04:03:43.958577 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958462    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a050ec-c2ba-4bf4-a625-36ce24f78a62" containerName="cleanup"
Oct 31 04:03:43.958577 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.958468    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="910e9857-1a61-41b9-a228-b39423e20d14" containerName="payload-pre-processing"
Oct 31 04:03:43.963154 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:43.963131    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.012869 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.012832    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c7585c-openshift-default-dockercfg-jnxt8\""
Oct 31 04:03:44.062342 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.062306    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"]
Oct 31 04:03:44.082055 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.082021    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-data\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.082245 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.082062    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-token\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.082245 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.082091    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istiod-ca-cert\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.082360 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.082238    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-envoy\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.082360 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.082273    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2pqq\" (UniqueName: \"kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-kube-api-access-t2pqq\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.082360 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.082297    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-certs\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.082525 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.082445    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-credential-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.082525 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.082489    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-podinfo\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.082680 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.082654    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.184164 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.184120    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-envoy\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.184164 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.184162    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2pqq\" (UniqueName: \"kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-kube-api-access-t2pqq\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.184416 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.184185    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-certs\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.184416 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.184220    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-credential-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.184416 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.184245    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-podinfo\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.184416 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.184305    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.184416 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.184384    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-data\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.184416 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.184413    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-token\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.184779 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.184453    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istiod-ca-cert\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.184841 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.184775    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-credential-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.184991 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.184965    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.184991 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.184978    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-certs\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.185149 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.185063    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-data\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.185353 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.185336    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istiod-ca-cert\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.186553 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.186534    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-envoy\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.186919 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.186898    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-podinfo\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.237923 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.237838    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-token\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.238124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.238102    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2pqq\" (UniqueName: \"kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-kube-api-access-t2pqq\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.276795 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.276760    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:44.701163 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.701005    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"]
Oct 31 04:03:44.703831 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:03:44.703799    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a458c7_fe27_4cef_9b32_9f69fe76c546.slice/crio-9b0f2bd3fcd09417c3859098dced4568e038533b52f295f33fa2880f3283cfb2 WatchSource:0}: Error finding container 9b0f2bd3fcd09417c3859098dced4568e038533b52f295f33fa2880f3283cfb2: Status 404 returned error can't find the container with id 9b0f2bd3fcd09417c3859098dced4568e038533b52f295f33fa2880f3283cfb2
Oct 31 04:03:44.706542 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.706507    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:03:44.706643 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.706581    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:03:44.706643 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.706610    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:03:44.914480 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.914444    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2" event={"ID":"d8a458c7-fe27-4cef-9b32-9f69fe76c546","Type":"ContainerStarted","Data":"f5f6dfc2b3c91dd8516ab86042957d05bbfaaa08388db9217a130ef3722b5dde"}
Oct 31 04:03:44.914663 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:44.914498    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2" event={"ID":"d8a458c7-fe27-4cef-9b32-9f69fe76c546","Type":"ContainerStarted","Data":"9b0f2bd3fcd09417c3859098dced4568e038533b52f295f33fa2880f3283cfb2"}
Oct 31 04:03:45.026613 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:45.026510    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2" podStartSLOduration=2.026493671 podStartE2EDuration="2.026493671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:45.026461164 +0000 UTC m=+3366.586940914" watchObservedRunningTime="2026-10-31 04:03:45.026493671 +0000 UTC m=+3366.586973402"
Oct 31 04:03:45.279980 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:45.279281    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:45.286086 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:45.286045    5415 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 31 04:03:45.286281 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:45.286114    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2" podUID="d8a458c7-fe27-4cef-9b32-9f69fe76c546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 31 04:03:46.023651 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.023596    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww" podUID="f5f16c8f-af5e-47a0-b925-cf44e3e4fef4" containerName="main" containerID="cri-o://d17171f2a7c8458ffbc77b02371b32b58a501912ec0b96c5c68b7df2ec3ea1d3" gracePeriod=45
Oct 31 04:03:46.172698 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.172642    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww" podUID="f5f16c8f-af5e-47a0-b925-cf44e3e4fef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.117:8000/ready\": dial tcp 10.134.0.117:8000: connect: connection refused"
Oct 31 04:03:46.204435 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:03:46.204387    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:03:46.282756 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.282656    5415 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 31 04:03:46.282990 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.282749    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2" podUID="d8a458c7-fe27-4cef-9b32-9f69fe76c546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 31 04:03:46.338751 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.338723    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 04:03:46.510741 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.510702    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-kube-api-access-wzsxn\" (UniqueName: \"kubernetes.io/projected/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-kube-api-access-wzsxn\") pod \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") "
Oct 31 04:03:46.510949 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.510752    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-home\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-home\") pod \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") "
Oct 31 04:03:46.510949 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.510778    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tmp-dir\") pod \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") "
Oct 31 04:03:46.511084 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.510966    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tls-certs\") pod \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") "
Oct 31 04:03:46.511084 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.511015    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-model-cache\") pod \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") "
Oct 31 04:03:46.511084 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.511079    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-dshm\") pod \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\" (UID: \"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4\") "
Oct 31 04:03:46.511245 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.511083    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-home" pod "f5f16c8f-af5e-47a0-b925-cf44e3e4fef4" (UID: "f5f16c8f-af5e-47a0-b925-cf44e3e4fef4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:03:46.511245 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.511178    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tmp-dir" pod "f5f16c8f-af5e-47a0-b925-cf44e3e4fef4" (UID: "f5f16c8f-af5e-47a0-b925-cf44e3e4fef4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:03:46.511379 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.511322    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-model-cache" pod "f5f16c8f-af5e-47a0-b925-cf44e3e4fef4" (UID: "f5f16c8f-af5e-47a0-b925-cf44e3e4fef4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:03:46.511543 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.511523    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:46.511543 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.511542    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:46.511680 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.511553    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:46.513139 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.513116    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-dshm" pod "f5f16c8f-af5e-47a0-b925-cf44e3e4fef4" (UID: "f5f16c8f-af5e-47a0-b925-cf44e3e4fef4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:03:46.513239 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.513161    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tls-certs" pod "f5f16c8f-af5e-47a0-b925-cf44e3e4fef4" (UID: "f5f16c8f-af5e-47a0-b925-cf44e3e4fef4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:03:46.513290 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.513240    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-kube-api-access-wzsxn" pod "f5f16c8f-af5e-47a0-b925-cf44e3e4fef4" (UID: "f5f16c8f-af5e-47a0-b925-cf44e3e4fef4"). InnerVolumeSpecName "kube-api-access-wzsxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:03:46.612189 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.612156    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:46.612189 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.612189    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzsxn\" (UniqueName: \"kubernetes.io/projected/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-kube-api-access-wzsxn\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:46.612400 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.612202    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:46.927623 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.927529    5415 generic.go:356] "Generic (PLEG): container finished" podID="f5f16c8f-af5e-47a0-b925-cf44e3e4fef4" containerID="d17171f2a7c8458ffbc77b02371b32b58a501912ec0b96c5c68b7df2ec3ea1d3" exitCode=0
Oct 31 04:03:46.927623 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.927583    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"
Oct 31 04:03:46.927623 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.927605    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww" event={"ID":"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4","Type":"ContainerDied","Data":"d17171f2a7c8458ffbc77b02371b32b58a501912ec0b96c5c68b7df2ec3ea1d3"}
Oct 31 04:03:46.927977 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.927653    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww" event={"ID":"f5f16c8f-af5e-47a0-b925-cf44e3e4fef4","Type":"ContainerDied","Data":"7243c72a9211daf8a860e148135f0f660589477003f294252fbfebb6164dda20"}
Oct 31 04:03:46.927977 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.927667    5415 scope.go:122] "RemoveContainer" containerID="d17171f2a7c8458ffbc77b02371b32b58a501912ec0b96c5c68b7df2ec3ea1d3"
Oct 31 04:03:46.936756 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.936719    5415 scope.go:122] "RemoveContainer" containerID="d17171f2a7c8458ffbc77b02371b32b58a501912ec0b96c5c68b7df2ec3ea1d3"
Oct 31 04:03:46.937055 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:03:46.937035    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17171f2a7c8458ffbc77b02371b32b58a501912ec0b96c5c68b7df2ec3ea1d3\": container with ID starting with d17171f2a7c8458ffbc77b02371b32b58a501912ec0b96c5c68b7df2ec3ea1d3 not found: ID does not exist" containerID="d17171f2a7c8458ffbc77b02371b32b58a501912ec0b96c5c68b7df2ec3ea1d3"
Oct 31 04:03:46.937137 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:46.937064    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17171f2a7c8458ffbc77b02371b32b58a501912ec0b96c5c68b7df2ec3ea1d3"} err="failed to get container status \"d17171f2a7c8458ffbc77b02371b32b58a501912ec0b96c5c68b7df2ec3ea1d3\": rpc error: code = NotFound desc = could not find container \"d17171f2a7c8458ffbc77b02371b32b58a501912ec0b96c5c68b7df2ec3ea1d3\": container with ID starting with d17171f2a7c8458ffbc77b02371b32b58a501912ec0b96c5c68b7df2ec3ea1d3 not found: ID does not exist"
Oct 31 04:03:47.083401 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.083368    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"]
Oct 31 04:03:47.130731 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.130690    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-861dfd/auth-test-model-861dfd-kserve-7bfbcdc4c7-7d5ww"]
Oct 31 04:03:47.278224 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.278139    5415 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 31 04:03:47.278224 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.278194    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2" podUID="d8a458c7-fe27-4cef-9b32-9f69fe76c546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 31 04:03:47.697976 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.697914    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"]
Oct 31 04:03:47.698493 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.698477    5415 cpu_manager.go:414] "containerMap: removing container" podUID="f5f16c8f-af5e-47a0-b925-cf44e3e4fef4" containerName="main"
Oct 31 04:03:47.698557 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.698495    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f16c8f-af5e-47a0-b925-cf44e3e4fef4" containerName="main"
Oct 31 04:03:47.698600 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.698591    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f16c8f-af5e-47a0-b925-cf44e3e4fef4" containerName="main"
Oct 31 04:03:47.703150 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.703132    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:47.752905 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.752853    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-e8fa21-serving-cert\""
Oct 31 04:03:47.761442 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.761390    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"]
Oct 31 04:03:47.826745 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.826704    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-metrics-certs\") pod \"maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:47.826952 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.826759    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb387fa5-b901-4dd9-8141-362071737055-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:47.827042 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.826938    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:47.827112 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.827086    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgldf\" (UniqueName: \"kubernetes.io/projected/eb387fa5-b901-4dd9-8141-362071737055-kube-api-access-xgldf\") pod \"maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:47.928590 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.928548    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgldf\" (UniqueName: \"kubernetes.io/projected/eb387fa5-b901-4dd9-8141-362071737055-kube-api-access-xgldf\") pod \"maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:47.928782 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.928648    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-metrics-certs\") pod \"maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:47.928782 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.928682    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb387fa5-b901-4dd9-8141-362071737055-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:47.928782 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.928737    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:47.929018 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:03:47.928985    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-serving-cert: secret "maas-api-e2e-extmod-a-e8fa21-serving-cert" not found
Oct 31 04:03:47.929092 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:03:47.929072    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-maas-api-tls podName:eb387fa5-b901-4dd9-8141-362071737055 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:48.429048479 +0000 UTC m=+3369.989528189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-maas-api-tls") pod "maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x" (UID: "eb387fa5-b901-4dd9-8141-362071737055") : secret "maas-api-e2e-extmod-a-e8fa21-serving-cert" not found
Oct 31 04:03:47.929401 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.929382    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb387fa5-b901-4dd9-8141-362071737055-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:47.931340 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.931311    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-metrics-certs\") pod \"maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:47.984601 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:47.984521    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgldf\" (UniqueName: \"kubernetes.io/projected/eb387fa5-b901-4dd9-8141-362071737055-kube-api-access-xgldf\") pod \"maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:48.278206 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:48.278118    5415 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 31 04:03:48.278206 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:48.278174    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2" podUID="d8a458c7-fe27-4cef-9b32-9f69fe76c546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 31 04:03:48.434859 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:48.434818    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:48.437374 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:48.437351    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:48.605691 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:48.605654    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f16c8f-af5e-47a0-b925-cf44e3e4fef4" path="/var/lib/kubelet/pods/f5f16c8f-af5e-47a0-b925-cf44e3e4fef4/volumes"
Oct 31 04:03:48.616432 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:48.616395    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:49.011867 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:49.011839    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"]
Oct 31 04:03:49.013905 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:03:49.013856    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb387fa5_b901_4dd9_8141_362071737055.slice/crio-b1eeb1b485904fcb3136aa36170437ac2edc081a89ce00df8f23e1ca5eea7fa6 WatchSource:0}: Error finding container b1eeb1b485904fcb3136aa36170437ac2edc081a89ce00df8f23e1ca5eea7fa6: Status 404 returned error can't find the container with id b1eeb1b485904fcb3136aa36170437ac2edc081a89ce00df8f23e1ca5eea7fa6
Oct 31 04:03:49.280366 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:49.280281    5415 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 31 04:03:49.280857 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:49.280450    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2" podUID="d8a458c7-fe27-4cef-9b32-9f69fe76c546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 31 04:03:49.946276 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:49.946241    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x" event={"ID":"eb387fa5-b901-4dd9-8141-362071737055","Type":"ContainerStarted","Data":"b1eeb1b485904fcb3136aa36170437ac2edc081a89ce00df8f23e1ca5eea7fa6"}
Oct 31 04:03:50.279964 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:50.279454    5415 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 31 04:03:50.279964 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:50.279538    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2" podUID="d8a458c7-fe27-4cef-9b32-9f69fe76c546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 31 04:03:50.955827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:50.955781    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x" event={"ID":"eb387fa5-b901-4dd9-8141-362071737055","Type":"ContainerStarted","Data":"bab5cb1c379e6d38559af7dbbda1ce3d5a8c7e06d63d0c71eeb8cec9931262fd"}
Oct 31 04:03:50.956346 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:50.955928    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:03:51.069218 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:51.069151    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x" podStartSLOduration=3.085079406 podStartE2EDuration="4.069129665s" totalImagesPullingTime="984.050259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:51.065771667 +0000 UTC m=+3372.626251409" watchObservedRunningTime="2026-10-31 04:03:51.069129665 +0000 UTC m=+3372.629609413"
Oct 31 04:03:51.282714 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:51.282638    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:51.282932 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:51.282908    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:51.283587 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:51.283567    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:03:56.966095 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:03:56.966066    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:04:16.175053 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:04:16.175013    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:04:35.913159 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:35.913118    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"]
Oct 31 04:04:35.917703 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:35.917675    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"
Oct 31 04:04:35.967200 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:35.967156    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-1f1314\""
Oct 31 04:04:36.014477 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:36.014441    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-1f1314-dockercfg-pmxhp\""
Oct 31 04:04:36.016588 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:36.016562    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"]
Oct 31 04:04:36.019508 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:36.019486    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zm8z\" (UniqueName: \"kubernetes.io/projected/a8787c50-6c33-4395-897c-e8629e2f5dc2-kube-api-access-2zm8z\") pod \"payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97\" (UID: \"a8787c50-6c33-4395-897c-e8629e2f5dc2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"
Oct 31 04:04:36.019570 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:36.019526    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8787c50-6c33-4395-897c-e8629e2f5dc2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97\" (UID: \"a8787c50-6c33-4395-897c-e8629e2f5dc2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"
Oct 31 04:04:36.120448 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:36.120411    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zm8z\" (UniqueName: \"kubernetes.io/projected/a8787c50-6c33-4395-897c-e8629e2f5dc2-kube-api-access-2zm8z\") pod \"payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97\" (UID: \"a8787c50-6c33-4395-897c-e8629e2f5dc2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"
Oct 31 04:04:36.120448 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:36.120458    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8787c50-6c33-4395-897c-e8629e2f5dc2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97\" (UID: \"a8787c50-6c33-4395-897c-e8629e2f5dc2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"
Oct 31 04:04:36.121160 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:36.121142    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8787c50-6c33-4395-897c-e8629e2f5dc2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97\" (UID: \"a8787c50-6c33-4395-897c-e8629e2f5dc2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"
Oct 31 04:04:36.175228 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:36.175154    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zm8z\" (UniqueName: \"kubernetes.io/projected/a8787c50-6c33-4395-897c-e8629e2f5dc2-kube-api-access-2zm8z\") pod \"payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97\" (UID: \"a8787c50-6c33-4395-897c-e8629e2f5dc2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"
Oct 31 04:04:36.233889 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:36.233848    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"
Oct 31 04:04:36.606589 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:36.606569    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"]
Oct 31 04:04:36.609595 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:04:36.609569    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8787c50_6c33_4395_897c_e8629e2f5dc2.slice/crio-eb12db3a878d041d291fffb74a1a6406cc706160346320191d88a9d56e775cd3 WatchSource:0}: Error finding container eb12db3a878d041d291fffb74a1a6406cc706160346320191d88a9d56e775cd3: Status 404 returned error can't find the container with id eb12db3a878d041d291fffb74a1a6406cc706160346320191d88a9d56e775cd3
Oct 31 04:04:37.217506 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:37.217474    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97" event={"ID":"a8787c50-6c33-4395-897c-e8629e2f5dc2","Type":"ContainerStarted","Data":"26b3409d18ec8b3d98423d80fd6152efcd5c240fdc4123a7e499182647302e49"}
Oct 31 04:04:37.217938 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:37.217524    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97" event={"ID":"a8787c50-6c33-4395-897c-e8629e2f5dc2","Type":"ContainerStarted","Data":"eb12db3a878d041d291fffb74a1a6406cc706160346320191d88a9d56e775cd3"}
Oct 31 04:04:37.217938 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:37.217679    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"
Oct 31 04:04:37.327533 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:37.327484    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97" podStartSLOduration=2.327467686 podStartE2EDuration="2.327467686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:37.326215842 +0000 UTC m=+3418.886695592" watchObservedRunningTime="2026-10-31 04:04:37.327467686 +0000 UTC m=+3418.887947415"
Oct 31 04:04:46.175186 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:04:46.175106    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:04:48.226098 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:04:48.226061    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"
Oct 31 04:05:16.177351 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:05:16.177313    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:05:23.953321 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:23.953284    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"]
Oct 31 04:05:23.957487 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:23.957464    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.054383 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.054343    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-23258c\"/\"kube-root-ca.crt\""
Oct 31 04:05:24.054383 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.054371    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-23258c\"/\"default-dockercfg-vz5bg\""
Oct 31 04:05:24.054612 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.054372    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-23258c\"/\"noten-model-23258c-kserve-self-signed-certs\""
Oct 31 04:05:24.054612 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.054396    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-23258c\"/\"openshift-service-ca.crt\""
Oct 31 04:05:24.056597 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.056578    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"]
Oct 31 04:05:24.083466 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.083428    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-tmp-dir\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.083466 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.083467    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-model-cache\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.083695 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.083601    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-home\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.083695 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.083630    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/010bbc9f-efb8-4023-a019-57d28ae27a94-tls-certs\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.083695 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.083672    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh98c\" (UniqueName: \"kubernetes.io/projected/010bbc9f-efb8-4023-a019-57d28ae27a94-kube-api-access-kh98c\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.083819 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.083750    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-dshm\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.184970 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.184921    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh98c\" (UniqueName: \"kubernetes.io/projected/010bbc9f-efb8-4023-a019-57d28ae27a94-kube-api-access-kh98c\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.185185 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.184982    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-dshm\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.185185 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.185085    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-tmp-dir\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.185185 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.185108    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-model-cache\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.185356 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.185209    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-home\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.185356 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.185240    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/010bbc9f-efb8-4023-a019-57d28ae27a94-tls-certs\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.185627 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.185589    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-model-cache\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.185790 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.185766    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-tmp-dir\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.185930 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.185812    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-home\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.187719 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.187694    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-dshm\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.188006 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.187984    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/010bbc9f-efb8-4023-a019-57d28ae27a94-tls-certs\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.239854 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.239778    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh98c\" (UniqueName: \"kubernetes.io/projected/010bbc9f-efb8-4023-a019-57d28ae27a94-kube-api-access-kh98c\") pod \"noten-model-23258c-kserve-56db974c89-cp74h\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") " pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.270175 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.270136    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:24.652593 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:24.652563    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"]
Oct 31 04:05:24.654435 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:05:24.654403    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod010bbc9f_efb8_4023_a019_57d28ae27a94.slice/crio-b1959a17a0280f294c8166b0aa0798390df6a664952d595215727a96e3750143 WatchSource:0}: Error finding container b1959a17a0280f294c8166b0aa0798390df6a664952d595215727a96e3750143: Status 404 returned error can't find the container with id b1959a17a0280f294c8166b0aa0798390df6a664952d595215727a96e3750143
Oct 31 04:05:25.510085 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:25.510047    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h" event={"ID":"010bbc9f-efb8-4023-a019-57d28ae27a94","Type":"ContainerStarted","Data":"495574ad97d9a647637e17c35da6c478981ad37a683c77cfc4d00944d193d656"}
Oct 31 04:05:25.510085 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:25.510094    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h" event={"ID":"010bbc9f-efb8-4023-a019-57d28ae27a94","Type":"ContainerStarted","Data":"b1959a17a0280f294c8166b0aa0798390df6a664952d595215727a96e3750143"}
Oct 31 04:05:25.620359 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:25.620306    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h" podStartSLOduration=2.620288105 podStartE2EDuration="2.620288105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:25.618099604 +0000 UTC m=+3467.178579335" watchObservedRunningTime="2026-10-31 04:05:25.620288105 +0000 UTC m=+3467.180767836"
Oct 31 04:05:26.271173 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:26.271137    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:34.271327 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:34.271294    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:34.283836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:34.283802    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:34.574767 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:05:34.574675    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:05:46.176693 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:05:46.176648    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:06:06.442524 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:06.442484    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"]
Oct 31 04:06:16.193317 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:06:16.193097    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:06:21.464676 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.464623    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h" podUID="010bbc9f-efb8-4023-a019-57d28ae27a94" containerName="main" containerID="cri-o://495574ad97d9a647637e17c35da6c478981ad37a683c77cfc4d00944d193d656" gracePeriod=45
Oct 31 04:06:21.564009 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.563967    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h" podUID="010bbc9f-efb8-4023-a019-57d28ae27a94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": dial tcp 10.134.0.125:8000: connect: connection refused"
Oct 31 04:06:21.743463 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.743428    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:06:21.833674 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.833633    5415 generic.go:356] "Generic (PLEG): container finished" podID="010bbc9f-efb8-4023-a019-57d28ae27a94" containerID="495574ad97d9a647637e17c35da6c478981ad37a683c77cfc4d00944d193d656" exitCode=0
Oct 31 04:06:21.833867 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.833687    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"
Oct 31 04:06:21.833867 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.833722    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h" event={"ID":"010bbc9f-efb8-4023-a019-57d28ae27a94","Type":"ContainerDied","Data":"495574ad97d9a647637e17c35da6c478981ad37a683c77cfc4d00944d193d656"}
Oct 31 04:06:21.833867 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.833753    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h" event={"ID":"010bbc9f-efb8-4023-a019-57d28ae27a94","Type":"ContainerDied","Data":"b1959a17a0280f294c8166b0aa0798390df6a664952d595215727a96e3750143"}
Oct 31 04:06:21.833867 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.833775    5415 scope.go:122] "RemoveContainer" containerID="495574ad97d9a647637e17c35da6c478981ad37a683c77cfc4d00944d193d656"
Oct 31 04:06:21.843270 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.843245    5415 scope.go:122] "RemoveContainer" containerID="495574ad97d9a647637e17c35da6c478981ad37a683c77cfc4d00944d193d656"
Oct 31 04:06:21.843402 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.843380    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010bbc9f-efb8-4023-a019-57d28ae27a94-kube-api-access-kh98c\" (UniqueName: \"kubernetes.io/projected/010bbc9f-efb8-4023-a019-57d28ae27a94-kube-api-access-kh98c\") pod \"010bbc9f-efb8-4023-a019-57d28ae27a94\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") "
Oct 31 04:06:21.843475 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.843458    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-model-cache\") pod \"010bbc9f-efb8-4023-a019-57d28ae27a94\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") "
Oct 31 04:06:21.843528 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.843505    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-home\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-home\") pod \"010bbc9f-efb8-4023-a019-57d28ae27a94\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") "
Oct 31 04:06:21.843568 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.843546    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-dshm\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-dshm\") pod \"010bbc9f-efb8-4023-a019-57d28ae27a94\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") "
Oct 31 04:06:21.843625 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.843577    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-tmp-dir\") pod \"010bbc9f-efb8-4023-a019-57d28ae27a94\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") "
Oct 31 04:06:21.843683 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.843648    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/010bbc9f-efb8-4023-a019-57d28ae27a94-tls-certs\" (UniqueName: \"kubernetes.io/secret/010bbc9f-efb8-4023-a019-57d28ae27a94-tls-certs\") pod \"010bbc9f-efb8-4023-a019-57d28ae27a94\" (UID: \"010bbc9f-efb8-4023-a019-57d28ae27a94\") "
Oct 31 04:06:21.843741 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:06:21.843663    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495574ad97d9a647637e17c35da6c478981ad37a683c77cfc4d00944d193d656\": container with ID starting with 495574ad97d9a647637e17c35da6c478981ad37a683c77cfc4d00944d193d656 not found: ID does not exist" containerID="495574ad97d9a647637e17c35da6c478981ad37a683c77cfc4d00944d193d656"
Oct 31 04:06:21.843741 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.843703    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495574ad97d9a647637e17c35da6c478981ad37a683c77cfc4d00944d193d656"} err="failed to get container status \"495574ad97d9a647637e17c35da6c478981ad37a683c77cfc4d00944d193d656\": rpc error: code = NotFound desc = could not find container \"495574ad97d9a647637e17c35da6c478981ad37a683c77cfc4d00944d193d656\": container with ID starting with 495574ad97d9a647637e17c35da6c478981ad37a683c77cfc4d00944d193d656 not found: ID does not exist"
Oct 31 04:06:21.843854 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.843807    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-model-cache" pod "010bbc9f-efb8-4023-a019-57d28ae27a94" (UID: "010bbc9f-efb8-4023-a019-57d28ae27a94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:21.843854 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.843816    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-home" pod "010bbc9f-efb8-4023-a019-57d28ae27a94" (UID: "010bbc9f-efb8-4023-a019-57d28ae27a94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:21.844240 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.844213    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-tmp-dir" pod "010bbc9f-efb8-4023-a019-57d28ae27a94" (UID: "010bbc9f-efb8-4023-a019-57d28ae27a94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:21.844336 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.844244    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:21.844336 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.844264    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:21.845751 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.845728    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-dshm" pod "010bbc9f-efb8-4023-a019-57d28ae27a94" (UID: "010bbc9f-efb8-4023-a019-57d28ae27a94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:21.845866 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.845847    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010bbc9f-efb8-4023-a019-57d28ae27a94-tls-certs" pod "010bbc9f-efb8-4023-a019-57d28ae27a94" (UID: "010bbc9f-efb8-4023-a019-57d28ae27a94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:06:21.845961 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.845895    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010bbc9f-efb8-4023-a019-57d28ae27a94-kube-api-access-kh98c" pod "010bbc9f-efb8-4023-a019-57d28ae27a94" (UID: "010bbc9f-efb8-4023-a019-57d28ae27a94"). InnerVolumeSpecName "kube-api-access-kh98c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:06:21.945433 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.945386    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/010bbc9f-efb8-4023-a019-57d28ae27a94-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:21.945433 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.945427    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh98c\" (UniqueName: \"kubernetes.io/projected/010bbc9f-efb8-4023-a019-57d28ae27a94-kube-api-access-kh98c\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:21.945433 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.945438    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:21.945679 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:21.945465    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/010bbc9f-efb8-4023-a019-57d28ae27a94-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:22.249120 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:22.249079    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"]
Oct 31 04:06:22.252723 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:22.252690    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-23258c/noten-model-23258c-kserve-56db974c89-cp74h"]
Oct 31 04:06:22.606543 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:22.606508    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010bbc9f-efb8-4023-a019-57d28ae27a94" path="/var/lib/kubelet/pods/010bbc9f-efb8-4023-a019-57d28ae27a94/volumes"
Oct 31 04:06:23.690301 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:23.690262    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"]
Oct 31 04:06:23.690822 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:23.690809    5415 cpu_manager.go:414] "containerMap: removing container" podUID="010bbc9f-efb8-4023-a019-57d28ae27a94" containerName="main"
Oct 31 04:06:23.690870 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:23.690823    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010bbc9f-efb8-4023-a019-57d28ae27a94" containerName="main"
Oct 31 04:06:23.690971 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:23.690959    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="010bbc9f-efb8-4023-a019-57d28ae27a94" containerName="main"
Oct 31 04:06:23.696101 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:23.696076    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"
Oct 31 04:06:23.745680 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:23.745651    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-cfbb5b\""
Oct 31 04:06:23.767221 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:23.767185    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335f5b25-8926-4e8f-9005-f713617d045a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg\" (UID: \"335f5b25-8926-4e8f-9005-f713617d045a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"
Oct 31 04:06:23.767406 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:23.767367    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqsc2\" (UniqueName: \"kubernetes.io/projected/335f5b25-8926-4e8f-9005-f713617d045a-kube-api-access-dqsc2\") pod \"payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg\" (UID: \"335f5b25-8926-4e8f-9005-f713617d045a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"
Oct 31 04:06:23.794324 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:23.794287    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"]
Oct 31 04:06:23.868557 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:23.868518    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqsc2\" (UniqueName: \"kubernetes.io/projected/335f5b25-8926-4e8f-9005-f713617d045a-kube-api-access-dqsc2\") pod \"payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg\" (UID: \"335f5b25-8926-4e8f-9005-f713617d045a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"
Oct 31 04:06:23.868777 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:23.868643    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335f5b25-8926-4e8f-9005-f713617d045a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg\" (UID: \"335f5b25-8926-4e8f-9005-f713617d045a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"
Oct 31 04:06:23.869372 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:23.869352    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335f5b25-8926-4e8f-9005-f713617d045a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg\" (UID: \"335f5b25-8926-4e8f-9005-f713617d045a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"
Oct 31 04:06:23.923159 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:23.923122    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqsc2\" (UniqueName: \"kubernetes.io/projected/335f5b25-8926-4e8f-9005-f713617d045a-kube-api-access-dqsc2\") pod \"payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg\" (UID: \"335f5b25-8926-4e8f-9005-f713617d045a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"
Oct 31 04:06:24.009787 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:24.009702    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"
Oct 31 04:06:24.380665 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:24.380639    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"]
Oct 31 04:06:24.383559 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:06:24.383348    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335f5b25_8926_4e8f_9005_f713617d045a.slice/crio-d09da5b81d8859a56c295d185ed6b77d8fb4c6b5e17c6d57cb8ec8fb55b3937b WatchSource:0}: Error finding container d09da5b81d8859a56c295d185ed6b77d8fb4c6b5e17c6d57cb8ec8fb55b3937b: Status 404 returned error can't find the container with id d09da5b81d8859a56c295d185ed6b77d8fb4c6b5e17c6d57cb8ec8fb55b3937b
Oct 31 04:06:24.853566 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:24.853531    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg" event={"ID":"335f5b25-8926-4e8f-9005-f713617d045a","Type":"ContainerStarted","Data":"6703c9ac60f639059d960eb58c02e4aa435bfb182d2bf3b7047f7e9294dc984d"}
Oct 31 04:06:24.854003 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:24.853582    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg" event={"ID":"335f5b25-8926-4e8f-9005-f713617d045a","Type":"ContainerStarted","Data":"d09da5b81d8859a56c295d185ed6b77d8fb4c6b5e17c6d57cb8ec8fb55b3937b"}
Oct 31 04:06:24.854003 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:24.853620    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"
Oct 31 04:06:24.961958 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:24.961900    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg" podStartSLOduration=1.961858888 podStartE2EDuration="1.961858888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:24.961174063 +0000 UTC m=+3526.521653815" watchObservedRunningTime="2026-10-31 04:06:24.961858888 +0000 UTC m=+3526.522338619"
Oct 31 04:06:35.862781 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:35.862746    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"
Oct 31 04:06:46.180548 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:06:46.180505    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:06:53.998424 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:53.998379    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"]
Oct 31 04:06:54.003614 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.003586    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.053476 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.053418    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0d5219-serving-cert\""
Oct 31 04:06:54.104039 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.103998    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"]
Oct 31 04:06:54.123681 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.123645    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krrvz\" (UniqueName: \"kubernetes.io/projected/5c9ed243-ce82-4a19-9b37-19ea45bea633-kube-api-access-krrvz\") pod \"maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.123872 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.123813    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9ed243-ce82-4a19-9b37-19ea45bea633-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.123872 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.123844    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-metrics-certs\") pod \"maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.123872 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.123865    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.224809 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.224772    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krrvz\" (UniqueName: \"kubernetes.io/projected/5c9ed243-ce82-4a19-9b37-19ea45bea633-kube-api-access-krrvz\") pod \"maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.225011 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.224915    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9ed243-ce82-4a19-9b37-19ea45bea633-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.225011 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.224949    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-metrics-certs\") pod \"maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.225011 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.224976    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.225158 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:06:54.225141    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-serving-cert: secret "maas-api-e2e-extmod-b-0d5219-serving-cert" not found
Oct 31 04:06:54.225244 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:06:54.225234    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-maas-api-tls podName:5c9ed243-ce82-4a19-9b37-19ea45bea633 nodeName:}" failed. No retries permitted until 2026-10-31 04:06:54.725208167 +0000 UTC m=+3556.285687881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-maas-api-tls") pod "maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j" (UID: "5c9ed243-ce82-4a19-9b37-19ea45bea633") : secret "maas-api-e2e-extmod-b-0d5219-serving-cert" not found
Oct 31 04:06:54.225692 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.225672    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9ed243-ce82-4a19-9b37-19ea45bea633-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.227589 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.227569    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-metrics-certs\") pod \"maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.280186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.280115    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krrvz\" (UniqueName: \"kubernetes.io/projected/5c9ed243-ce82-4a19-9b37-19ea45bea633-kube-api-access-krrvz\") pod \"maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.730327 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.730282    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.732730 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.732697    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:54.917357 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:54.917318    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:55.289163 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:55.289134    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"]
Oct 31 04:06:55.291130 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:06:55.291087    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9ed243_ce82_4a19_9b37_19ea45bea633.slice/crio-d0ea57f593971ad141ae22924167623d74fa5f79f0a53768acbdf496daf17d40 WatchSource:0}: Error finding container d0ea57f593971ad141ae22924167623d74fa5f79f0a53768acbdf496daf17d40: Status 404 returned error can't find the container with id d0ea57f593971ad141ae22924167623d74fa5f79f0a53768acbdf496daf17d40
Oct 31 04:06:56.039870 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:56.039827    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j" event={"ID":"5c9ed243-ce82-4a19-9b37-19ea45bea633","Type":"ContainerStarted","Data":"d0ea57f593971ad141ae22924167623d74fa5f79f0a53768acbdf496daf17d40"}
Oct 31 04:06:57.047606 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:57.047562    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j" event={"ID":"5c9ed243-ce82-4a19-9b37-19ea45bea633","Type":"ContainerStarted","Data":"17bbe1c423c372943039f1b8397ababa8c734d5d36c04f26586a5591345754f7"}
Oct 31 04:06:57.048065 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:57.047673    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:06:57.162522 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:06:57.162466    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j" podStartSLOduration=3.297731472 podStartE2EDuration="4.162453437s" totalImagesPullingTime="864.721965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:57.161505485 +0000 UTC m=+3558.721985216" watchObservedRunningTime="2026-10-31 04:06:57.162453437 +0000 UTC m=+3558.722933168"
Oct 31 04:07:03.059345 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:03.059313    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:07:16.177266 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:07:16.177217    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:07:46.177626 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:07:46.177544    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:07:53.928484 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:53.928447    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"]
Oct 31 04:07:53.932449 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:53.932423    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:53.982217 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:53.982175    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-cfbb5b-serving-cert\""
Oct 31 04:07:54.020563 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.020515    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:54.020765 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.020656    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:54.020765 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.020744    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw8mt\" (UniqueName: \"kubernetes.io/projected/5116f65c-2424-4d40-9227-6849db407611-kube-api-access-cw8mt\") pod \"maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:54.020904 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.020869    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5116f65c-2424-4d40-9227-6849db407611-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:54.031075 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.031041    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"]
Oct 31 04:07:54.122087 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.122044    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5116f65c-2424-4d40-9227-6849db407611-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:54.122319 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.122138    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:54.122319 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.122232    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:54.122448 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:07:54.122309    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-serving-cert: secret "maas-api-e2e-shared-b-w0-cfbb5b-serving-cert" not found
Oct 31 04:07:54.122448 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:07:54.122391    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-maas-api-tls podName:5116f65c-2424-4d40-9227-6849db407611 nodeName:}" failed. No retries permitted until 2026-10-31 04:07:54.622366908 +0000 UTC m=+3616.182846638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-maas-api-tls") pod "maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh" (UID: "5116f65c-2424-4d40-9227-6849db407611") : secret "maas-api-e2e-shared-b-w0-cfbb5b-serving-cert" not found
Oct 31 04:07:54.122448 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.122315    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8mt\" (UniqueName: \"kubernetes.io/projected/5116f65c-2424-4d40-9227-6849db407611-kube-api-access-cw8mt\") pod \"maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:54.122895 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.122855    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5116f65c-2424-4d40-9227-6849db407611-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:54.124844 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.124820    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:54.176789 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.176759    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8mt\" (UniqueName: \"kubernetes.io/projected/5116f65c-2424-4d40-9227-6849db407611-kube-api-access-cw8mt\") pod \"maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:54.629100 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.629058    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:54.631840 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.631814    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:54.846601 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:54.846561    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:55.216801 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:55.216773    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"]
Oct 31 04:07:55.219575 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:07:55.219540    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5116f65c_2424_4d40_9227_6849db407611.slice/crio-cbdaf5c18eaef04ab67d4bc12d87bea0a1bf6e75a40986b40b8f0a91b79c1510 WatchSource:0}: Error finding container cbdaf5c18eaef04ab67d4bc12d87bea0a1bf6e75a40986b40b8f0a91b79c1510: Status 404 returned error can't find the container with id cbdaf5c18eaef04ab67d4bc12d87bea0a1bf6e75a40986b40b8f0a91b79c1510
Oct 31 04:07:55.383498 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:55.383449    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh" event={"ID":"5116f65c-2424-4d40-9227-6849db407611","Type":"ContainerStarted","Data":"cbdaf5c18eaef04ab67d4bc12d87bea0a1bf6e75a40986b40b8f0a91b79c1510"}
Oct 31 04:07:56.391843 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:56.391808    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh" event={"ID":"5116f65c-2424-4d40-9227-6849db407611","Type":"ContainerStarted","Data":"4beba143f3a4b5a4a7113416a467712daddef210d398dc2a50f765587c334234"}
Oct 31 04:07:56.392289 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:56.391930    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:07:56.500793 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:07:56.500738    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh" podStartSLOduration=2.609739865 podStartE2EDuration="3.500723052s" totalImagesPullingTime="890.983187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:07:56.499808282 +0000 UTC m=+3618.060288014" watchObservedRunningTime="2026-10-31 04:07:56.500723052 +0000 UTC m=+3618.061202783"
Oct 31 04:08:02.404401 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:02.404366    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:08:16.177351 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:08:16.177310    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:08:46.176704 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:08:46.176663    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:08:50.736914 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:50.736858    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"]
Oct 31 04:08:50.737379 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:50.737115    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk" podUID="dccd2ac0-5607-4a92-8a94-d32108a1b0b5" containerName="maas-api" containerID="cri-o://b10710ef38cdc7e58a99d5803b6e0d0143eaf95ff3543c5c5b96408f5a7252fc" gracePeriod=30
Oct 31 04:08:51.037091 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.037056    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:08:51.189763 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.189724    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-kube-api-access-fl44d\" (UniqueName: \"kubernetes.io/projected/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-kube-api-access-fl44d\") pod \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") "
Oct 31 04:08:51.189977 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.189897    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-maas-api-tls\") pod \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") "
Oct 31 04:08:51.190034 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.190023    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-openshift-service-ca\") pod \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") "
Oct 31 04:08:51.190077 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.190055    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-metrics-certs\") pod \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\" (UID: \"dccd2ac0-5607-4a92-8a94-d32108a1b0b5\") "
Oct 31 04:08:51.191358 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.191314    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-openshift-service-ca" pod "dccd2ac0-5607-4a92-8a94-d32108a1b0b5" (UID: "dccd2ac0-5607-4a92-8a94-d32108a1b0b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:08:51.200926 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.192914    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-metrics-certs" pod "dccd2ac0-5607-4a92-8a94-d32108a1b0b5" (UID: "dccd2ac0-5607-4a92-8a94-d32108a1b0b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:08:51.200926 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.193635    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-kube-api-access-fl44d" pod "dccd2ac0-5607-4a92-8a94-d32108a1b0b5" (UID: "dccd2ac0-5607-4a92-8a94-d32108a1b0b5"). InnerVolumeSpecName "kube-api-access-fl44d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:08:51.200926 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.194317    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-maas-api-tls" pod "dccd2ac0-5607-4a92-8a94-d32108a1b0b5" (UID: "dccd2ac0-5607-4a92-8a94-d32108a1b0b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:08:51.292382 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.292296    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.292382 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.292330    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.292382 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.292341    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.292382 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.292350    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl44d\" (UniqueName: \"kubernetes.io/projected/dccd2ac0-5607-4a92-8a94-d32108a1b0b5-kube-api-access-fl44d\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.710534 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.710501    5415 generic.go:356] "Generic (PLEG): container finished" podID="dccd2ac0-5607-4a92-8a94-d32108a1b0b5" containerID="b10710ef38cdc7e58a99d5803b6e0d0143eaf95ff3543c5c5b96408f5a7252fc" exitCode=0
Oct 31 04:08:51.710757 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.710547    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"
Oct 31 04:08:51.710757 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.710590    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk" event={"ID":"dccd2ac0-5607-4a92-8a94-d32108a1b0b5","Type":"ContainerDied","Data":"b10710ef38cdc7e58a99d5803b6e0d0143eaf95ff3543c5c5b96408f5a7252fc"}
Oct 31 04:08:51.710757 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.710657    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk" event={"ID":"dccd2ac0-5607-4a92-8a94-d32108a1b0b5","Type":"ContainerDied","Data":"0c3331c0f8c98cf74e4f423187a387ad404de067f48ec04a03702c214b88a2e2"}
Oct 31 04:08:51.710757 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.710691    5415 scope.go:122] "RemoveContainer" containerID="b10710ef38cdc7e58a99d5803b6e0d0143eaf95ff3543c5c5b96408f5a7252fc"
Oct 31 04:08:51.721760 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.721739    5415 scope.go:122] "RemoveContainer" containerID="b10710ef38cdc7e58a99d5803b6e0d0143eaf95ff3543c5c5b96408f5a7252fc"
Oct 31 04:08:51.722083 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:08:51.722062    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10710ef38cdc7e58a99d5803b6e0d0143eaf95ff3543c5c5b96408f5a7252fc\": container with ID starting with b10710ef38cdc7e58a99d5803b6e0d0143eaf95ff3543c5c5b96408f5a7252fc not found: ID does not exist" containerID="b10710ef38cdc7e58a99d5803b6e0d0143eaf95ff3543c5c5b96408f5a7252fc"
Oct 31 04:08:51.722180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.722090    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10710ef38cdc7e58a99d5803b6e0d0143eaf95ff3543c5c5b96408f5a7252fc"} err="failed to get container status \"b10710ef38cdc7e58a99d5803b6e0d0143eaf95ff3543c5c5b96408f5a7252fc\": rpc error: code = NotFound desc = could not find container \"b10710ef38cdc7e58a99d5803b6e0d0143eaf95ff3543c5c5b96408f5a7252fc\": container with ID starting with b10710ef38cdc7e58a99d5803b6e0d0143eaf95ff3543c5c5b96408f5a7252fc not found: ID does not exist"
Oct 31 04:08:51.869493 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.869456    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"]
Oct 31 04:08:51.915613 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:51.915577    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7585c-6475d9776f-t4dfk"]
Oct 31 04:08:52.605555 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:08:52.605521    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dccd2ac0-5607-4a92-8a94-d32108a1b0b5" path="/var/lib/kubelet/pods/dccd2ac0-5607-4a92-8a94-d32108a1b0b5/volumes"
Oct 31 04:09:00.184745 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:00.184708    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"]
Oct 31 04:09:00.185283 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:00.184954    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2" podUID="d8a458c7-fe27-4cef-9b32-9f69fe76c546" containerName="istio-proxy" containerID="cri-o://f5f6dfc2b3c91dd8516ab86042957d05bbfaaa08388db9217a130ef3722b5dde" gracePeriod=30
Oct 31 04:09:05.450257 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.450184    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:09:05.554784 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.554745    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istiod-ca-cert\") pod \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") "
Oct 31 04:09:05.555002 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.554818    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-credential-socket\") pod \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") "
Oct 31 04:09:05.555002 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.554928    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-podinfo\") pod \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") "
Oct 31 04:09:05.555002 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.554954    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-socket\") pod \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") "
Oct 31 04:09:05.555108 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555034    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-envoy\") pod \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") "
Oct 31 04:09:05.555108 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555063    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-data\") pod \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") "
Oct 31 04:09:05.555108 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555091    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-kube-api-access-t2pqq\" (UniqueName: \"kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-kube-api-access-t2pqq\") pod \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") "
Oct 31 04:09:05.555272 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555124    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-token\" (UniqueName: \"kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-token\") pod \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") "
Oct 31 04:09:05.555272 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555156    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-certs\") pod \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\" (UID: \"d8a458c7-fe27-4cef-9b32-9f69fe76c546\") "
Oct 31 04:09:05.555272 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555177    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-credential-socket" pod "d8a458c7-fe27-4cef-9b32-9f69fe76c546" (UID: "d8a458c7-fe27-4cef-9b32-9f69fe76c546"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:09:05.555272 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555228    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istiod-ca-cert" pod "d8a458c7-fe27-4cef-9b32-9f69fe76c546" (UID: "d8a458c7-fe27-4cef-9b32-9f69fe76c546"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:05.555474 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555341    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-socket" pod "d8a458c7-fe27-4cef-9b32-9f69fe76c546" (UID: "d8a458c7-fe27-4cef-9b32-9f69fe76c546"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:09:05.555474 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555357    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-data" pod "d8a458c7-fe27-4cef-9b32-9f69fe76c546" (UID: "d8a458c7-fe27-4cef-9b32-9f69fe76c546"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:09:05.555571 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555534    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-certs" pod "d8a458c7-fe27-4cef-9b32-9f69fe76c546" (UID: "d8a458c7-fe27-4cef-9b32-9f69fe76c546"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:09:05.555715 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555700    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.555825 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555720    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.555825 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555733    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.555825 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555748    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.555825 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.555761    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.557449 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.557416    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-kube-api-access-t2pqq" pod "d8a458c7-fe27-4cef-9b32-9f69fe76c546" (UID: "d8a458c7-fe27-4cef-9b32-9f69fe76c546"). InnerVolumeSpecName "kube-api-access-t2pqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:05.557449 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.557435    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-envoy" pod "d8a458c7-fe27-4cef-9b32-9f69fe76c546" (UID: "d8a458c7-fe27-4cef-9b32-9f69fe76c546"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:09:05.557657 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.557640    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-token" pod "d8a458c7-fe27-4cef-9b32-9f69fe76c546" (UID: "d8a458c7-fe27-4cef-9b32-9f69fe76c546"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:05.557718 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.557647    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-podinfo" pod "d8a458c7-fe27-4cef-9b32-9f69fe76c546" (UID: "d8a458c7-fe27-4cef-9b32-9f69fe76c546"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:09:05.657251 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.657214    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.657251 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.657245    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2pqq\" (UniqueName: \"kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-kube-api-access-t2pqq\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.657251 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.657258    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.657599 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.657266    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a458c7-fe27-4cef-9b32-9f69fe76c546-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.790614 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.790520    5415 generic.go:356] "Generic (PLEG): container finished" podID="d8a458c7-fe27-4cef-9b32-9f69fe76c546" containerID="f5f6dfc2b3c91dd8516ab86042957d05bbfaaa08388db9217a130ef3722b5dde" exitCode=0
Oct 31 04:09:05.790614 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.790574    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"
Oct 31 04:09:05.790614 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.790594    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2" event={"ID":"d8a458c7-fe27-4cef-9b32-9f69fe76c546","Type":"ContainerDied","Data":"f5f6dfc2b3c91dd8516ab86042957d05bbfaaa08388db9217a130ef3722b5dde"}
Oct 31 04:09:05.790858 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.790638    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2" event={"ID":"d8a458c7-fe27-4cef-9b32-9f69fe76c546","Type":"ContainerDied","Data":"9b0f2bd3fcd09417c3859098dced4568e038533b52f295f33fa2880f3283cfb2"}
Oct 31 04:09:05.790858 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.790659    5415 scope.go:122] "RemoveContainer" containerID="f5f6dfc2b3c91dd8516ab86042957d05bbfaaa08388db9217a130ef3722b5dde"
Oct 31 04:09:05.801761 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.801738    5415 scope.go:122] "RemoveContainer" containerID="f5f6dfc2b3c91dd8516ab86042957d05bbfaaa08388db9217a130ef3722b5dde"
Oct 31 04:09:05.802127 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:09:05.802103    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f6dfc2b3c91dd8516ab86042957d05bbfaaa08388db9217a130ef3722b5dde\": container with ID starting with f5f6dfc2b3c91dd8516ab86042957d05bbfaaa08388db9217a130ef3722b5dde not found: ID does not exist" containerID="f5f6dfc2b3c91dd8516ab86042957d05bbfaaa08388db9217a130ef3722b5dde"
Oct 31 04:09:05.802180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.802139    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f6dfc2b3c91dd8516ab86042957d05bbfaaa08388db9217a130ef3722b5dde"} err="failed to get container status \"f5f6dfc2b3c91dd8516ab86042957d05bbfaaa08388db9217a130ef3722b5dde\": rpc error: code = NotFound desc = could not find container \"f5f6dfc2b3c91dd8516ab86042957d05bbfaaa08388db9217a130ef3722b5dde\": container with ID starting with f5f6dfc2b3c91dd8516ab86042957d05bbfaaa08388db9217a130ef3722b5dde not found: ID does not exist"
Oct 31 04:09:05.946573 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.946535    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"]
Oct 31 04:09:05.950985 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:05.950946    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-vqxf2"]
Oct 31 04:09:06.608978 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:06.608934    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a458c7-fe27-4cef-9b32-9f69fe76c546" path="/var/lib/kubelet/pods/d8a458c7-fe27-4cef-9b32-9f69fe76c546/volumes"
Oct 31 04:09:07.561414 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:07.561367    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"]
Oct 31 04:09:12.561870 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:12.561796    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97" podUID="a8787c50-6c33-4395-897c-e8629e2f5dc2" containerName="payload-processing" containerID="cri-o://26b3409d18ec8b3d98423d80fd6152efcd5c240fdc4123a7e499182647302e49" gracePeriod=25
Oct 31 04:09:12.845254 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:12.845219    5415 generic.go:356] "Generic (PLEG): container finished" podID="a8787c50-6c33-4395-897c-e8629e2f5dc2" containerID="26b3409d18ec8b3d98423d80fd6152efcd5c240fdc4123a7e499182647302e49" exitCode=0
Oct 31 04:09:12.845436 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:12.845276    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97" event={"ID":"a8787c50-6c33-4395-897c-e8629e2f5dc2","Type":"ContainerDied","Data":"26b3409d18ec8b3d98423d80fd6152efcd5c240fdc4123a7e499182647302e49"}
Oct 31 04:09:12.845436 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:12.845315    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97" event={"ID":"a8787c50-6c33-4395-897c-e8629e2f5dc2","Type":"ContainerDied","Data":"eb12db3a878d041d291fffb74a1a6406cc706160346320191d88a9d56e775cd3"}
Oct 31 04:09:12.845436 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:12.845331    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb12db3a878d041d291fffb74a1a6406cc706160346320191d88a9d56e775cd3"
Oct 31 04:09:12.853067 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:12.853028    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"
Oct 31 04:09:12.952745 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:12.952712    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8787c50-6c33-4395-897c-e8629e2f5dc2-kube-api-access-2zm8z\" (UniqueName: \"kubernetes.io/projected/a8787c50-6c33-4395-897c-e8629e2f5dc2-kube-api-access-2zm8z\") pod \"a8787c50-6c33-4395-897c-e8629e2f5dc2\" (UID: \"a8787c50-6c33-4395-897c-e8629e2f5dc2\") "
Oct 31 04:09:12.953012 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:12.952848    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8787c50-6c33-4395-897c-e8629e2f5dc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8787c50-6c33-4395-897c-e8629e2f5dc2-plugins-config-volume\") pod \"a8787c50-6c33-4395-897c-e8629e2f5dc2\" (UID: \"a8787c50-6c33-4395-897c-e8629e2f5dc2\") "
Oct 31 04:09:12.953231 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:12.953208    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8787c50-6c33-4395-897c-e8629e2f5dc2-plugins-config-volume" pod "a8787c50-6c33-4395-897c-e8629e2f5dc2" (UID: "a8787c50-6c33-4395-897c-e8629e2f5dc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:12.954970 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:12.954947    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8787c50-6c33-4395-897c-e8629e2f5dc2-kube-api-access-2zm8z" pod "a8787c50-6c33-4395-897c-e8629e2f5dc2" (UID: "a8787c50-6c33-4395-897c-e8629e2f5dc2"). InnerVolumeSpecName "kube-api-access-2zm8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:13.054241 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:13.054203    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8787c50-6c33-4395-897c-e8629e2f5dc2-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:13.054241 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:13.054238    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zm8z\" (UniqueName: \"kubernetes.io/projected/a8787c50-6c33-4395-897c-e8629e2f5dc2-kube-api-access-2zm8z\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:13.850573 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:13.850537    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"
Oct 31 04:09:14.005202 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:14.005168    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"]
Oct 31 04:09:14.007006 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:14.006980    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1f1314-7985cfb565-4sz97"]
Oct 31 04:09:14.605983 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:14.605947    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8787c50-6c33-4395-897c-e8629e2f5dc2" path="/var/lib/kubelet/pods/a8787c50-6c33-4395-897c-e8629e2f5dc2/volumes"
Oct 31 04:09:16.177046 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:09:16.176969    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:09:28.102849 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.102809    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"]
Oct 31 04:09:28.103490 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.103468    5415 cpu_manager.go:414] "containerMap: removing container" podUID="d8a458c7-fe27-4cef-9b32-9f69fe76c546" containerName="istio-proxy"
Oct 31 04:09:28.103537 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.103492    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a458c7-fe27-4cef-9b32-9f69fe76c546" containerName="istio-proxy"
Oct 31 04:09:28.103576 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.103537    5415 cpu_manager.go:414] "containerMap: removing container" podUID="dccd2ac0-5607-4a92-8a94-d32108a1b0b5" containerName="maas-api"
Oct 31 04:09:28.103576 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.103542    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccd2ac0-5607-4a92-8a94-d32108a1b0b5" containerName="maas-api"
Oct 31 04:09:28.103576 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.103552    5415 cpu_manager.go:414] "containerMap: removing container" podUID="a8787c50-6c33-4395-897c-e8629e2f5dc2" containerName="payload-processing"
Oct 31 04:09:28.103576 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.103558    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8787c50-6c33-4395-897c-e8629e2f5dc2" containerName="payload-processing"
Oct 31 04:09:28.103695 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.103637    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccd2ac0-5607-4a92-8a94-d32108a1b0b5" containerName="maas-api"
Oct 31 04:09:28.103695 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.103644    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8787c50-6c33-4395-897c-e8629e2f5dc2" containerName="payload-processing"
Oct 31 04:09:28.103695 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.103654    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a458c7-fe27-4cef-9b32-9f69fe76c546" containerName="istio-proxy"
Oct 31 04:09:28.108034 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.108016    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.157368 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.157326    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-96js4\""
Oct 31 04:09:28.227748 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.227705    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.227963 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.227772    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpd7m\" (UniqueName: \"kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-kube-api-access-dpd7m\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.227963 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.227843    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.228076 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.227978    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31e83da-c659-4596-ac00-c7e4369911bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.228076 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.228042    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.228183 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.228073    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31e83da-c659-4596-ac00-c7e4369911bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.228183 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.228140    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.228268 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.228199    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.228268 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.228257    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.319567 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.319534    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"]
Oct 31 04:09:28.329894 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.329825    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.330070 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.329926    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.330070 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.329986    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpd7m\" (UniqueName: \"kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-kube-api-access-dpd7m\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.330070 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.330048    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.330210 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.330127    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31e83da-c659-4596-ac00-c7e4369911bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.330210 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.330163    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.330210 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.330204    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31e83da-c659-4596-ac00-c7e4369911bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.330320 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.330240    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.330320 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.330290    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.330826 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.330798    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.331957 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.331923    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31e83da-c659-4596-ac00-c7e4369911bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.332378 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.332358    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.332756 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.332734    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.333015 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.332984    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.333653 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.333607    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.333934 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.333914    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31e83da-c659-4596-ac00-c7e4369911bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.386000 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.385904    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpd7m\" (UniqueName: \"kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-kube-api-access-dpd7m\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.386770 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.386746    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ppc6s\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.422112 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.422079    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:28.841970 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.841944    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"]
Oct 31 04:09:28.843913 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:09:28.843868    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31e83da_c659_4596_ac00_c7e4369911bf.slice/crio-b2c3f554a12f0c7817a4282595d4362d4958c7e0e79081973567264a5cd0af78 WatchSource:0}: Error finding container b2c3f554a12f0c7817a4282595d4362d4958c7e0e79081973567264a5cd0af78: Status 404 returned error can't find the container with id b2c3f554a12f0c7817a4282595d4362d4958c7e0e79081973567264a5cd0af78
Oct 31 04:09:28.846753 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.846717    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:09:28.846858 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.846811    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:09:28.846946 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.846894    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:09:28.938113 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.938076    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" event={"ID":"b31e83da-c659-4596-ac00-c7e4369911bf","Type":"ContainerStarted","Data":"e982eb46517652004c8173bd7b7a453c7cb0c2600abe2dfe39643ec480451d11"}
Oct 31 04:09:28.938319 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:28.938165    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" event={"ID":"b31e83da-c659-4596-ac00-c7e4369911bf","Type":"ContainerStarted","Data":"b2c3f554a12f0c7817a4282595d4362d4958c7e0e79081973567264a5cd0af78"}
Oct 31 04:09:29.048390 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:29.048335    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" podStartSLOduration=1.048321169 podStartE2EDuration="1.048321169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:29.047427668 +0000 UTC m=+3710.607907425" watchObservedRunningTime="2026-10-31 04:09:29.048321169 +0000 UTC m=+3710.608800900"
Oct 31 04:09:29.422545 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:29.422497    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:29.424154 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:29.424128    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ppc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 04:09:29.424284 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:29.424197    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 04:09:30.423082 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:30.423042    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ppc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 04:09:30.423512 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:30.423131    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 04:09:31.422456 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:31.422419    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ppc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 04:09:31.422661 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:31.422476    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 04:09:32.422798 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:32.422757    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ppc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 04:09:32.423245 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:32.422823    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 04:09:33.423070 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:33.423034    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ppc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 04:09:33.423510 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:33.423097    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 04:09:34.422789 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:34.422745    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ppc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 04:09:34.423043 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:34.422809    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 04:09:35.422707 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:35.422668    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ppc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 04:09:35.423164 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:35.422727    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 04:09:36.422728 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:36.422689    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ppc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 04:09:36.423175 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:36.422754    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 04:09:38.464094 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:38.464043    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ppc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:09:38.464665 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:38.464101    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:09:39.514158 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:39.514106    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ppc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:09:39.514703 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:39.514191    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:09:39.578726 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:39.578691    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:39.579648 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:39.579621    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:39.580521 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:39.580495    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:09:44.702195 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:44.702151    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"]
Oct 31 04:09:44.702673 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:44.702454    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x" podUID="eb387fa5-b901-4dd9-8141-362071737055" containerName="maas-api" containerID="cri-o://bab5cb1c379e6d38559af7dbbda1ce3d5a8c7e06d63d0c71eeb8cec9931262fd" gracePeriod=30
Oct 31 04:09:44.991004 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:44.990973    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:09:45.055213 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.055171    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb387fa5-b901-4dd9-8141-362071737055-kube-api-access-xgldf\" (UniqueName: \"kubernetes.io/projected/eb387fa5-b901-4dd9-8141-362071737055-kube-api-access-xgldf\") pod \"eb387fa5-b901-4dd9-8141-362071737055\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") "
Oct 31 04:09:45.055392 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.055267    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-maas-api-tls\") pod \"eb387fa5-b901-4dd9-8141-362071737055\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") "
Oct 31 04:09:45.055480 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.055392    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb387fa5-b901-4dd9-8141-362071737055-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb387fa5-b901-4dd9-8141-362071737055-openshift-service-ca\") pod \"eb387fa5-b901-4dd9-8141-362071737055\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") "
Oct 31 04:09:45.055539 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.055485    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-metrics-certs\") pod \"eb387fa5-b901-4dd9-8141-362071737055\" (UID: \"eb387fa5-b901-4dd9-8141-362071737055\") "
Oct 31 04:09:45.055977 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.055948    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb387fa5-b901-4dd9-8141-362071737055-openshift-service-ca" pod "eb387fa5-b901-4dd9-8141-362071737055" (UID: "eb387fa5-b901-4dd9-8141-362071737055"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:45.058045 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.058012    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb387fa5-b901-4dd9-8141-362071737055-kube-api-access-xgldf" pod "eb387fa5-b901-4dd9-8141-362071737055" (UID: "eb387fa5-b901-4dd9-8141-362071737055"). InnerVolumeSpecName "kube-api-access-xgldf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:45.058233 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.058201    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-maas-api-tls" pod "eb387fa5-b901-4dd9-8141-362071737055" (UID: "eb387fa5-b901-4dd9-8141-362071737055"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:09:45.058352 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.058257    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-metrics-certs" pod "eb387fa5-b901-4dd9-8141-362071737055" (UID: "eb387fa5-b901-4dd9-8141-362071737055"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:09:45.069678 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.068943    5415 generic.go:356] "Generic (PLEG): container finished" podID="eb387fa5-b901-4dd9-8141-362071737055" containerID="bab5cb1c379e6d38559af7dbbda1ce3d5a8c7e06d63d0c71eeb8cec9931262fd" exitCode=0
Oct 31 04:09:45.069678 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.068991    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x" event={"ID":"eb387fa5-b901-4dd9-8141-362071737055","Type":"ContainerDied","Data":"bab5cb1c379e6d38559af7dbbda1ce3d5a8c7e06d63d0c71eeb8cec9931262fd"}
Oct 31 04:09:45.069678 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.069003    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"
Oct 31 04:09:45.069678 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.069043    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x" event={"ID":"eb387fa5-b901-4dd9-8141-362071737055","Type":"ContainerDied","Data":"b1eeb1b485904fcb3136aa36170437ac2edc081a89ce00df8f23e1ca5eea7fa6"}
Oct 31 04:09:45.069678 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.069304    5415 scope.go:122] "RemoveContainer" containerID="bab5cb1c379e6d38559af7dbbda1ce3d5a8c7e06d63d0c71eeb8cec9931262fd"
Oct 31 04:09:45.088313 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.088287    5415 scope.go:122] "RemoveContainer" containerID="bab5cb1c379e6d38559af7dbbda1ce3d5a8c7e06d63d0c71eeb8cec9931262fd"
Oct 31 04:09:45.088718 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:09:45.088686    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab5cb1c379e6d38559af7dbbda1ce3d5a8c7e06d63d0c71eeb8cec9931262fd\": container with ID starting with bab5cb1c379e6d38559af7dbbda1ce3d5a8c7e06d63d0c71eeb8cec9931262fd not found: ID does not exist" containerID="bab5cb1c379e6d38559af7dbbda1ce3d5a8c7e06d63d0c71eeb8cec9931262fd"
Oct 31 04:09:45.088807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.088735    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab5cb1c379e6d38559af7dbbda1ce3d5a8c7e06d63d0c71eeb8cec9931262fd"} err="failed to get container status \"bab5cb1c379e6d38559af7dbbda1ce3d5a8c7e06d63d0c71eeb8cec9931262fd\": rpc error: code = NotFound desc = could not find container \"bab5cb1c379e6d38559af7dbbda1ce3d5a8c7e06d63d0c71eeb8cec9931262fd\": container with ID starting with bab5cb1c379e6d38559af7dbbda1ce3d5a8c7e06d63d0c71eeb8cec9931262fd not found: ID does not exist"
Oct 31 04:09:45.156827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.156789    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:45.156827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.156822    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgldf\" (UniqueName: \"kubernetes.io/projected/eb387fa5-b901-4dd9-8141-362071737055-kube-api-access-xgldf\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:45.156827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.156834    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb387fa5-b901-4dd9-8141-362071737055-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:45.157183 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.156843    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb387fa5-b901-4dd9-8141-362071737055-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:45.226332 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.226251    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"]
Oct 31 04:09:45.232665 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:45.232630    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8fa21-8674bf5699-2hk2x"]
Oct 31 04:09:46.175799 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:09:46.175761    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:09:46.605822 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:09:46.605789    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb387fa5-b901-4dd9-8141-362071737055" path="/var/lib/kubelet/pods/eb387fa5-b901-4dd9-8141-362071737055/volumes"
Oct 31 04:10:02.039086 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.039042    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"]
Oct 31 04:10:02.039662 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.039299    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j" podUID="5c9ed243-ce82-4a19-9b37-19ea45bea633" containerName="maas-api" containerID="cri-o://17bbe1c423c372943039f1b8397ababa8c734d5d36c04f26586a5591345754f7" gracePeriod=30
Oct 31 04:10:02.170157 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.170119    5415 generic.go:356] "Generic (PLEG): container finished" podID="5c9ed243-ce82-4a19-9b37-19ea45bea633" containerID="17bbe1c423c372943039f1b8397ababa8c734d5d36c04f26586a5591345754f7" exitCode=0
Oct 31 04:10:02.170325 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.170194    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j" event={"ID":"5c9ed243-ce82-4a19-9b37-19ea45bea633","Type":"ContainerDied","Data":"17bbe1c423c372943039f1b8397ababa8c734d5d36c04f26586a5591345754f7"}
Oct 31 04:10:02.373859 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.373831    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:10:02.462405 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.462354    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c9ed243-ce82-4a19-9b37-19ea45bea633-kube-api-access-krrvz\" (UniqueName: \"kubernetes.io/projected/5c9ed243-ce82-4a19-9b37-19ea45bea633-kube-api-access-krrvz\") pod \"5c9ed243-ce82-4a19-9b37-19ea45bea633\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") "
Oct 31 04:10:02.462595 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.462416    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-maas-api-tls\") pod \"5c9ed243-ce82-4a19-9b37-19ea45bea633\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") "
Oct 31 04:10:02.462595 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.462574    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-metrics-certs\") pod \"5c9ed243-ce82-4a19-9b37-19ea45bea633\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") "
Oct 31 04:10:02.464710 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.464679    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-metrics-certs" pod "5c9ed243-ce82-4a19-9b37-19ea45bea633" (UID: "5c9ed243-ce82-4a19-9b37-19ea45bea633"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:10:02.465162 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.465140    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9ed243-ce82-4a19-9b37-19ea45bea633-kube-api-access-krrvz" pod "5c9ed243-ce82-4a19-9b37-19ea45bea633" (UID: "5c9ed243-ce82-4a19-9b37-19ea45bea633"). InnerVolumeSpecName "kube-api-access-krrvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:02.465267 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.465210    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-maas-api-tls" pod "5c9ed243-ce82-4a19-9b37-19ea45bea633" (UID: "5c9ed243-ce82-4a19-9b37-19ea45bea633"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:10:02.563838 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.563804    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c9ed243-ce82-4a19-9b37-19ea45bea633-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9ed243-ce82-4a19-9b37-19ea45bea633-openshift-service-ca\") pod \"5c9ed243-ce82-4a19-9b37-19ea45bea633\" (UID: \"5c9ed243-ce82-4a19-9b37-19ea45bea633\") "
Oct 31 04:10:02.564150 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.564134    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krrvz\" (UniqueName: \"kubernetes.io/projected/5c9ed243-ce82-4a19-9b37-19ea45bea633-kube-api-access-krrvz\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:02.564197 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.564152    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:02.564197 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.564163    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c9ed243-ce82-4a19-9b37-19ea45bea633-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:02.564264 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.564207    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c9ed243-ce82-4a19-9b37-19ea45bea633-openshift-service-ca" pod "5c9ed243-ce82-4a19-9b37-19ea45bea633" (UID: "5c9ed243-ce82-4a19-9b37-19ea45bea633"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:02.664800 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:02.664762    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9ed243-ce82-4a19-9b37-19ea45bea633-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:03.178982 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:03.178932    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j" event={"ID":"5c9ed243-ce82-4a19-9b37-19ea45bea633","Type":"ContainerDied","Data":"d0ea57f593971ad141ae22924167623d74fa5f79f0a53768acbdf496daf17d40"}
Oct 31 04:10:03.178982 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:03.178966    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"
Oct 31 04:10:03.179568 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:03.179027    5415 scope.go:122] "RemoveContainer" containerID="17bbe1c423c372943039f1b8397ababa8c734d5d36c04f26586a5591345754f7"
Oct 31 04:10:03.335582 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:03.335546    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"]
Oct 31 04:10:03.340345 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:03.340311    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d5219-858dc9c44d-l5k4j"]
Oct 31 04:10:04.615628 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:04.615581    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c9ed243-ce82-4a19-9b37-19ea45bea633" path="/var/lib/kubelet/pods/5c9ed243-ce82-4a19-9b37-19ea45bea633/volumes"
Oct 31 04:10:13.451980 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.451942    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"]
Oct 31 04:10:13.452454 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.452441    5415 cpu_manager.go:414] "containerMap: removing container" podUID="eb387fa5-b901-4dd9-8141-362071737055" containerName="maas-api"
Oct 31 04:10:13.452497 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.452455    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb387fa5-b901-4dd9-8141-362071737055" containerName="maas-api"
Oct 31 04:10:13.452497 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.452473    5415 cpu_manager.go:414] "containerMap: removing container" podUID="5c9ed243-ce82-4a19-9b37-19ea45bea633" containerName="maas-api"
Oct 31 04:10:13.452497 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.452478    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c9ed243-ce82-4a19-9b37-19ea45bea633" containerName="maas-api"
Oct 31 04:10:13.452596 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.452573    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c9ed243-ce82-4a19-9b37-19ea45bea633" containerName="maas-api"
Oct 31 04:10:13.452596 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.452588    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb387fa5-b901-4dd9-8141-362071737055" containerName="maas-api"
Oct 31 04:10:13.455827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.455808    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.554654 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.554615    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"]
Oct 31 04:10:13.587658 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.587624    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpqd\" (UniqueName: \"kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-kube-api-access-ffpqd\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.587836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.587665    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.587836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.587687    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.587836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.587823    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.587970 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.587946    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.588004 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.587976    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.588004 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.587997    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.588071 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.588033    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.588107 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.588081    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83393f8e-2253-42a6-9fc0-25e5cc488155-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.689317 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.689271    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.689317 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.689317    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83393f8e-2253-42a6-9fc0-25e5cc488155-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.689626 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.689408    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpqd\" (UniqueName: \"kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-kube-api-access-ffpqd\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.689626 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.689432    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.689626 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.689449    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.689626 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.689480    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.689626 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.689576    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.689626 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.689616    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.690008 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.689649    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.690008 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.689812    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.690116 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.690027    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.690213 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.690178    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.690370 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.690348    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.690506 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.690485    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83393f8e-2253-42a6-9fc0-25e5cc488155-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.692007 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.691987    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.692204 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.692185    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.743844 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.743764    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.744019 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.743900    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpqd\" (UniqueName: \"kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-kube-api-access-ffpqd\") pod \"maas-default-gateway-openshift-default-6886b96f9-jdrnv\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:13.769906 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:13.769842    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:14.191520 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:14.191496    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"]
Oct 31 04:10:14.195622 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:10:14.195587    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83393f8e_2253_42a6_9fc0_25e5cc488155.slice/crio-789e8f90871834cace8aa950dcfc92e621e9125a588405888b9964a5b64172a2 WatchSource:0}: Error finding container 789e8f90871834cace8aa950dcfc92e621e9125a588405888b9964a5b64172a2: Status 404 returned error can't find the container with id 789e8f90871834cace8aa950dcfc92e621e9125a588405888b9964a5b64172a2
Oct 31 04:10:14.199171 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:14.199126    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:10:14.199337 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:14.199227    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:10:14.199337 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:14.199275    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:10:14.242593 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:14.242559    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv" event={"ID":"83393f8e-2253-42a6-9fc0-25e5cc488155","Type":"ContainerStarted","Data":"789e8f90871834cace8aa950dcfc92e621e9125a588405888b9964a5b64172a2"}
Oct 31 04:10:15.275272 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:15.275219    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv" event={"ID":"83393f8e-2253-42a6-9fc0-25e5cc488155","Type":"ContainerStarted","Data":"57b46e09d8bb75d7a75cdd452fdb683781c5a0416f9cc626ff93ddb8f16dd980"}
Oct 31 04:10:15.437257 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:15.437200    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv" podStartSLOduration=2.43718492 podStartE2EDuration="2.43718492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:15.435758922 +0000 UTC m=+3756.996238679" watchObservedRunningTime="2026-10-31 04:10:15.43718492 +0000 UTC m=+3756.997664739"
Oct 31 04:10:15.771903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:15.770906    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:15.772818 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:15.772783    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jdrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 04:10:15.773001 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:15.772850    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv" podUID="83393f8e-2253-42a6-9fc0-25e5cc488155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 04:10:16.198241 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:10:16.198202    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:10:16.770322 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:16.770282    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jdrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 04:10:16.770992 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:16.770960    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv" podUID="83393f8e-2253-42a6-9fc0-25e5cc488155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 04:10:17.772527 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:17.772488    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jdrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 04:10:17.773058 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:17.772636    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv" podUID="83393f8e-2253-42a6-9fc0-25e5cc488155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 04:10:18.770904 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:18.770356    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jdrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 04:10:18.770904 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:18.770442    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv" podUID="83393f8e-2253-42a6-9fc0-25e5cc488155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 04:10:19.770592 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:19.770535    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jdrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 04:10:19.771308 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:19.771271    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv" podUID="83393f8e-2253-42a6-9fc0-25e5cc488155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 04:10:20.771116 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:20.771073    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jdrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 04:10:20.771643 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:20.771160    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv" podUID="83393f8e-2253-42a6-9fc0-25e5cc488155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 04:10:21.770943 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:21.770905    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jdrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 04:10:21.771129 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:21.770976    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv" podUID="83393f8e-2253-42a6-9fc0-25e5cc488155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 04:10:22.874078 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:22.874042    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:22.874615 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:22.874430    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:22.875783 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:22.875757    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:10:39.556544 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:39.556510    5415 scope.go:122] "RemoveContainer" containerID="26b3409d18ec8b3d98423d80fd6152efcd5c240fdc4123a7e499182647302e49"
Oct 31 04:10:43.671028 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.670988    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"]
Oct 31 04:10:43.676023 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.675995    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.797685 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.797647    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96326289-4481-4634-b123-e93c01e0c1dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.797888 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.797702    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.797888 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.797820    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.797984 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.797933    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.798029 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.797999    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.798064 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.798027    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm85x\" (UniqueName: \"kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-kube-api-access-gm85x\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.798103 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.798074    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96326289-4481-4634-b123-e93c01e0c1dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.798144 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.798129    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.798176 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.798161    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.886736 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.886696    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"]
Oct 31 04:10:43.899115 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.899077    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.899318 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.899143    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm85x\" (UniqueName: \"kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-kube-api-access-gm85x\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.899318 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.899176    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96326289-4481-4634-b123-e93c01e0c1dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.899318 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.899254    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.899318 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.899283    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.899579 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.899320    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96326289-4481-4634-b123-e93c01e0c1dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.899579 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.899369    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.899579 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.899418    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.899579 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.899529    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.899579 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.899550    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.899865 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.899844    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.900029 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.899968    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.900277 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.900257    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.900415 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.900400    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96326289-4481-4634-b123-e93c01e0c1dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.902629 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.902603    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.903074 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.903054    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96326289-4481-4634-b123-e93c01e0c1dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.954562 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.954484    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.954710 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.954621    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm85x\" (UniqueName: \"kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-kube-api-access-gm85x\") pod \"maas-default-gateway-openshift-default-6886b96f9-vh5lb\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:43.989663 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:43.989623    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:44.418496 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:44.418456    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"]
Oct 31 04:10:44.456608 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:10:44.456570    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96326289_4481_4634_b123_e93c01e0c1dc.slice/crio-74e7af8bd7cda8fc93d087f51cdb2a1d2e805b24fca3ee97ada441586d43417c WatchSource:0}: Error finding container 74e7af8bd7cda8fc93d087f51cdb2a1d2e805b24fca3ee97ada441586d43417c: Status 404 returned error can't find the container with id 74e7af8bd7cda8fc93d087f51cdb2a1d2e805b24fca3ee97ada441586d43417c
Oct 31 04:10:44.459762 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:44.459724    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:10:44.459915 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:44.459815    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:10:44.459915 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:44.459863    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:10:45.476001 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:45.475917    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb" event={"ID":"96326289-4481-4634-b123-e93c01e0c1dc","Type":"ContainerStarted","Data":"05cd0de54992dd32b08024ba0883d15822afc56a781dcf1e27a0cbb3641e8133"}
Oct 31 04:10:45.476001 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:45.475978    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb" event={"ID":"96326289-4481-4634-b123-e93c01e0c1dc","Type":"ContainerStarted","Data":"74e7af8bd7cda8fc93d087f51cdb2a1d2e805b24fca3ee97ada441586d43417c"}
Oct 31 04:10:45.637312 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:45.635358    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb" podStartSLOduration=2.635338576 podStartE2EDuration="2.635338576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:45.633800167 +0000 UTC m=+3787.194279915" watchObservedRunningTime="2026-10-31 04:10:45.635338576 +0000 UTC m=+3787.195818312"
Oct 31 04:10:45.990030 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:45.989992    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:45.992050 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:45.992018    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vh5lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:10:45.992190 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:45.992080    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb" podUID="96326289-4481-4634-b123-e93c01e0c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:10:46.205054 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:10:46.205015    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:10:46.990421 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:46.990382    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vh5lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:10:46.990851 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:46.990445    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb" podUID="96326289-4481-4634-b123-e93c01e0c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:10:47.990396 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:47.990359    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vh5lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:10:47.990595 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:47.990416    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb" podUID="96326289-4481-4634-b123-e93c01e0c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:10:48.641767 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:48.641727    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"]
Oct 31 04:10:48.803004 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:48.802960    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"]
Oct 31 04:10:48.990707 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:48.990613    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vh5lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:10:48.990707 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:48.990676    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb" podUID="96326289-4481-4634-b123-e93c01e0c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:10:49.501153 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:49.501114    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh" podUID="5116f65c-2424-4d40-9227-6849db407611" containerName="maas-api" containerID="cri-o://4beba143f3a4b5a4a7113416a467712daddef210d398dc2a50f765587c334234" gracePeriod=30
Oct 31 04:10:49.808733 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:49.808707    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:10:49.906545 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:49.905434    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5116f65c-2424-4d40-9227-6849db407611-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5116f65c-2424-4d40-9227-6849db407611-openshift-service-ca\") pod \"5116f65c-2424-4d40-9227-6849db407611\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") "
Oct 31 04:10:49.906545 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:49.905535    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-maas-api-tls\") pod \"5116f65c-2424-4d40-9227-6849db407611\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") "
Oct 31 04:10:49.906545 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:49.905581    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-metrics-certs\") pod \"5116f65c-2424-4d40-9227-6849db407611\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") "
Oct 31 04:10:49.906545 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:49.905674    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5116f65c-2424-4d40-9227-6849db407611-kube-api-access-cw8mt\" (UniqueName: \"kubernetes.io/projected/5116f65c-2424-4d40-9227-6849db407611-kube-api-access-cw8mt\") pod \"5116f65c-2424-4d40-9227-6849db407611\" (UID: \"5116f65c-2424-4d40-9227-6849db407611\") "
Oct 31 04:10:49.909467 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:49.909427    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5116f65c-2424-4d40-9227-6849db407611-openshift-service-ca" pod "5116f65c-2424-4d40-9227-6849db407611" (UID: "5116f65c-2424-4d40-9227-6849db407611"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:49.913559 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:49.913498    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5116f65c-2424-4d40-9227-6849db407611-kube-api-access-cw8mt" pod "5116f65c-2424-4d40-9227-6849db407611" (UID: "5116f65c-2424-4d40-9227-6849db407611"). InnerVolumeSpecName "kube-api-access-cw8mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:49.914210 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:49.914157    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-maas-api-tls" pod "5116f65c-2424-4d40-9227-6849db407611" (UID: "5116f65c-2424-4d40-9227-6849db407611"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:10:49.915504 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:49.915477    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-metrics-certs" pod "5116f65c-2424-4d40-9227-6849db407611" (UID: "5116f65c-2424-4d40-9227-6849db407611"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:10:49.990511 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:49.990475    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vh5lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:10:49.990708 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:49.990539    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb" podUID="96326289-4481-4634-b123-e93c01e0c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:10:50.007501 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.007466    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5116f65c-2424-4d40-9227-6849db407611-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:50.007501 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.007501    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:50.007990 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.007518    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5116f65c-2424-4d40-9227-6849db407611-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:50.007990 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.007547    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw8mt\" (UniqueName: \"kubernetes.io/projected/5116f65c-2424-4d40-9227-6849db407611-kube-api-access-cw8mt\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:50.507619 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.507585    5415 generic.go:356] "Generic (PLEG): container finished" podID="5116f65c-2424-4d40-9227-6849db407611" containerID="4beba143f3a4b5a4a7113416a467712daddef210d398dc2a50f765587c334234" exitCode=0
Oct 31 04:10:50.507829 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.507687    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh" event={"ID":"5116f65c-2424-4d40-9227-6849db407611","Type":"ContainerDied","Data":"4beba143f3a4b5a4a7113416a467712daddef210d398dc2a50f765587c334234"}
Oct 31 04:10:50.507829 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.507726    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh" event={"ID":"5116f65c-2424-4d40-9227-6849db407611","Type":"ContainerDied","Data":"cbdaf5c18eaef04ab67d4bc12d87bea0a1bf6e75a40986b40b8f0a91b79c1510"}
Oct 31 04:10:50.507829 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.507747    5415 scope.go:122] "RemoveContainer" containerID="4beba143f3a4b5a4a7113416a467712daddef210d398dc2a50f765587c334234"
Oct 31 04:10:50.507957 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.507872    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"
Oct 31 04:10:50.518991 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.518959    5415 scope.go:122] "RemoveContainer" containerID="4beba143f3a4b5a4a7113416a467712daddef210d398dc2a50f765587c334234"
Oct 31 04:10:50.519342 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:10:50.519321    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4beba143f3a4b5a4a7113416a467712daddef210d398dc2a50f765587c334234\": container with ID starting with 4beba143f3a4b5a4a7113416a467712daddef210d398dc2a50f765587c334234 not found: ID does not exist" containerID="4beba143f3a4b5a4a7113416a467712daddef210d398dc2a50f765587c334234"
Oct 31 04:10:50.519412 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.519353    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4beba143f3a4b5a4a7113416a467712daddef210d398dc2a50f765587c334234"} err="failed to get container status \"4beba143f3a4b5a4a7113416a467712daddef210d398dc2a50f765587c334234\": rpc error: code = NotFound desc = could not find container \"4beba143f3a4b5a4a7113416a467712daddef210d398dc2a50f765587c334234\": container with ID starting with 4beba143f3a4b5a4a7113416a467712daddef210d398dc2a50f765587c334234 not found: ID does not exist"
Oct 31 04:10:50.664297 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.664259    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"]
Oct 31 04:10:50.711757 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.711712    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-cfbb5b-784655dfc7-qfznh"]
Oct 31 04:10:50.990845 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.990811    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vh5lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:10:50.991063 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:50.990866    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb" podUID="96326289-4481-4634-b123-e93c01e0c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:10:51.990600 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:51.990563    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vh5lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:10:51.991053 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:51.990624    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb" podUID="96326289-4481-4634-b123-e93c01e0c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:10:52.606051 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:52.606018    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5116f65c-2424-4d40-9227-6849db407611" path="/var/lib/kubelet/pods/5116f65c-2424-4d40-9227-6849db407611/volumes"
Oct 31 04:10:53.017308 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:53.017207    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:53.017811 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:53.017614    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:53.020115 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:53.020085    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:10:53.642839 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:53.642792    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg" podUID="335f5b25-8926-4e8f-9005-f713617d045a" containerName="payload-pre-processing" containerID="cri-o://6703c9ac60f639059d960eb58c02e4aa435bfb182d2bf3b7047f7e9294dc984d" gracePeriod=25
Oct 31 04:10:53.905198 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:53.905122    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"
Oct 31 04:10:53.960526 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:53.960489    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335f5b25-8926-4e8f-9005-f713617d045a-kube-api-access-dqsc2\" (UniqueName: \"kubernetes.io/projected/335f5b25-8926-4e8f-9005-f713617d045a-kube-api-access-dqsc2\") pod \"335f5b25-8926-4e8f-9005-f713617d045a\" (UID: \"335f5b25-8926-4e8f-9005-f713617d045a\") "
Oct 31 04:10:53.960723 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:53.960629    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335f5b25-8926-4e8f-9005-f713617d045a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335f5b25-8926-4e8f-9005-f713617d045a-plugins-config-volume\") pod \"335f5b25-8926-4e8f-9005-f713617d045a\" (UID: \"335f5b25-8926-4e8f-9005-f713617d045a\") "
Oct 31 04:10:53.961106 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:53.961072    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335f5b25-8926-4e8f-9005-f713617d045a-plugins-config-volume" pod "335f5b25-8926-4e8f-9005-f713617d045a" (UID: "335f5b25-8926-4e8f-9005-f713617d045a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:53.962690 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:53.962670    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335f5b25-8926-4e8f-9005-f713617d045a-kube-api-access-dqsc2" pod "335f5b25-8926-4e8f-9005-f713617d045a" (UID: "335f5b25-8926-4e8f-9005-f713617d045a"). InnerVolumeSpecName "kube-api-access-dqsc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:54.062327 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:54.062290    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335f5b25-8926-4e8f-9005-f713617d045a-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:54.062327 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:54.062325    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqsc2\" (UniqueName: \"kubernetes.io/projected/335f5b25-8926-4e8f-9005-f713617d045a-kube-api-access-dqsc2\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:54.534556 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:54.534524    5415 generic.go:356] "Generic (PLEG): container finished" podID="335f5b25-8926-4e8f-9005-f713617d045a" containerID="6703c9ac60f639059d960eb58c02e4aa435bfb182d2bf3b7047f7e9294dc984d" exitCode=0
Oct 31 04:10:54.534731 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:54.534568    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"
Oct 31 04:10:54.534731 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:54.534592    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg" event={"ID":"335f5b25-8926-4e8f-9005-f713617d045a","Type":"ContainerDied","Data":"6703c9ac60f639059d960eb58c02e4aa435bfb182d2bf3b7047f7e9294dc984d"}
Oct 31 04:10:54.534731 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:54.534647    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg" event={"ID":"335f5b25-8926-4e8f-9005-f713617d045a","Type":"ContainerDied","Data":"d09da5b81d8859a56c295d185ed6b77d8fb4c6b5e17c6d57cb8ec8fb55b3937b"}
Oct 31 04:10:54.534731 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:54.534672    5415 scope.go:122] "RemoveContainer" containerID="6703c9ac60f639059d960eb58c02e4aa435bfb182d2bf3b7047f7e9294dc984d"
Oct 31 04:10:54.545220 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:54.545190    5415 scope.go:122] "RemoveContainer" containerID="6703c9ac60f639059d960eb58c02e4aa435bfb182d2bf3b7047f7e9294dc984d"
Oct 31 04:10:54.545643 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:10:54.545619    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6703c9ac60f639059d960eb58c02e4aa435bfb182d2bf3b7047f7e9294dc984d\": container with ID starting with 6703c9ac60f639059d960eb58c02e4aa435bfb182d2bf3b7047f7e9294dc984d not found: ID does not exist" containerID="6703c9ac60f639059d960eb58c02e4aa435bfb182d2bf3b7047f7e9294dc984d"
Oct 31 04:10:54.545745 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:54.545656    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6703c9ac60f639059d960eb58c02e4aa435bfb182d2bf3b7047f7e9294dc984d"} err="failed to get container status \"6703c9ac60f639059d960eb58c02e4aa435bfb182d2bf3b7047f7e9294dc984d\": rpc error: code = NotFound desc = could not find container \"6703c9ac60f639059d960eb58c02e4aa435bfb182d2bf3b7047f7e9294dc984d\": container with ID starting with 6703c9ac60f639059d960eb58c02e4aa435bfb182d2bf3b7047f7e9294dc984d not found: ID does not exist"
Oct 31 04:10:54.689136 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:54.689095    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"]
Oct 31 04:10:54.736294 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:54.736259    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-cfbb5b-7985d9687-wwfsg"]
Oct 31 04:10:56.605832 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:10:56.605794    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335f5b25-8926-4e8f-9005-f713617d045a" path="/var/lib/kubelet/pods/335f5b25-8926-4e8f-9005-f713617d045a/volumes"
Oct 31 04:11:08.167453 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.167412    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"]
Oct 31 04:11:08.168160 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.168141    5415 cpu_manager.go:414] "containerMap: removing container" podUID="5116f65c-2424-4d40-9227-6849db407611" containerName="maas-api"
Oct 31 04:11:08.168219 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.168162    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5116f65c-2424-4d40-9227-6849db407611" containerName="maas-api"
Oct 31 04:11:08.168219 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.168192    5415 cpu_manager.go:414] "containerMap: removing container" podUID="335f5b25-8926-4e8f-9005-f713617d045a" containerName="payload-pre-processing"
Oct 31 04:11:08.168219 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.168197    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335f5b25-8926-4e8f-9005-f713617d045a" containerName="payload-pre-processing"
Oct 31 04:11:08.168312 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.168299    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="335f5b25-8926-4e8f-9005-f713617d045a" containerName="payload-pre-processing"
Oct 31 04:11:08.168312 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.168308    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="5116f65c-2424-4d40-9227-6849db407611" containerName="maas-api"
Oct 31 04:11:08.172927 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.172902    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"
Oct 31 04:11:08.270401 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.270366    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d247af\""
Oct 31 04:11:08.270955 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.270865    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d247af-dockercfg-jsr7w\""
Oct 31 04:11:08.274608 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.274580    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"]
Oct 31 04:11:08.333988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.333953    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97gr5\" (UniqueName: \"kubernetes.io/projected/a8b3f000-ea82-4cb7-9d3a-8582790da783-kube-api-access-97gr5\") pod \"payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss\" (UID: \"a8b3f000-ea82-4cb7-9d3a-8582790da783\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"
Oct 31 04:11:08.334180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.334101    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b3f000-ea82-4cb7-9d3a-8582790da783-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss\" (UID: \"a8b3f000-ea82-4cb7-9d3a-8582790da783\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"
Oct 31 04:11:08.435487 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.435388    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b3f000-ea82-4cb7-9d3a-8582790da783-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss\" (UID: \"a8b3f000-ea82-4cb7-9d3a-8582790da783\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"
Oct 31 04:11:08.435669 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.435527    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97gr5\" (UniqueName: \"kubernetes.io/projected/a8b3f000-ea82-4cb7-9d3a-8582790da783-kube-api-access-97gr5\") pod \"payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss\" (UID: \"a8b3f000-ea82-4cb7-9d3a-8582790da783\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"
Oct 31 04:11:08.436191 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.436169    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b3f000-ea82-4cb7-9d3a-8582790da783-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss\" (UID: \"a8b3f000-ea82-4cb7-9d3a-8582790da783\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"
Oct 31 04:11:08.492528 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.492497    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97gr5\" (UniqueName: \"kubernetes.io/projected/a8b3f000-ea82-4cb7-9d3a-8582790da783-kube-api-access-97gr5\") pod \"payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss\" (UID: \"a8b3f000-ea82-4cb7-9d3a-8582790da783\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"
Oct 31 04:11:08.786920 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:08.786802    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"
Oct 31 04:11:09.154155 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:09.154123    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"]
Oct 31 04:11:09.156702 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:11:09.156651    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b3f000_ea82_4cb7_9d3a_8582790da783.slice/crio-bb834d30d64c676c0c84634bc863b659835f6f1c11a53ba648e6f3d0d285f74a WatchSource:0}: Error finding container bb834d30d64c676c0c84634bc863b659835f6f1c11a53ba648e6f3d0d285f74a: Status 404 returned error can't find the container with id bb834d30d64c676c0c84634bc863b659835f6f1c11a53ba648e6f3d0d285f74a
Oct 31 04:11:09.636702 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:09.636658    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss" event={"ID":"a8b3f000-ea82-4cb7-9d3a-8582790da783","Type":"ContainerStarted","Data":"df3e1782cbf4de3e2df83a20eb74fe37f24a02636f807d7ccd78a318157791bf"}
Oct 31 04:11:09.637249 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:09.636723    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"
Oct 31 04:11:09.637249 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:09.636742    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss" event={"ID":"a8b3f000-ea82-4cb7-9d3a-8582790da783","Type":"ContainerStarted","Data":"bb834d30d64c676c0c84634bc863b659835f6f1c11a53ba648e6f3d0d285f74a"}
Oct 31 04:11:09.743912 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:09.743813    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss" podStartSLOduration=1.7437925459999999 podStartE2EDuration="1.743792546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:09.743443878 +0000 UTC m=+3811.303923627" watchObservedRunningTime="2026-10-31 04:11:09.743792546 +0000 UTC m=+3811.304272279"
Oct 31 04:11:16.176222 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:11:16.176183    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:11:20.647730 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:20.647701    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"
Oct 31 04:11:33.486394 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:33.486355    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"]
Oct 31 04:11:33.491007 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:33.490984    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"
Oct 31 04:11:33.540663 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:33.540629    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e2a54d\""
Oct 31 04:11:33.587641 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:33.587605    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e2a54d-dockercfg-bdccn\""
Oct 31 04:11:33.589991 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:33.589958    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"]
Oct 31 04:11:33.633899 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:33.633664    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvtxb\" (UniqueName: \"kubernetes.io/projected/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-kube-api-access-jvtxb\") pod \"payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm\" (UID: \"10f36a0c-b261-4862-b6fa-f8c3747fe6a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"
Oct 31 04:11:33.636099 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:33.636057    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm\" (UID: \"10f36a0c-b261-4862-b6fa-f8c3747fe6a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"
Oct 31 04:11:33.737129 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:33.737026    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtxb\" (UniqueName: \"kubernetes.io/projected/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-kube-api-access-jvtxb\") pod \"payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm\" (UID: \"10f36a0c-b261-4862-b6fa-f8c3747fe6a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"
Oct 31 04:11:33.737336 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:33.737162    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm\" (UID: \"10f36a0c-b261-4862-b6fa-f8c3747fe6a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"
Oct 31 04:11:33.737956 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:33.737921    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm\" (UID: \"10f36a0c-b261-4862-b6fa-f8c3747fe6a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"
Oct 31 04:11:33.791124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:33.791091    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvtxb\" (UniqueName: \"kubernetes.io/projected/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-kube-api-access-jvtxb\") pod \"payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm\" (UID: \"10f36a0c-b261-4862-b6fa-f8c3747fe6a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"
Oct 31 04:11:33.805959 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:33.805924    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"
Oct 31 04:11:34.176151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:34.176118    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"]
Oct 31 04:11:34.179312 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:11:34.179281    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f36a0c_b261_4862_b6fa_f8c3747fe6a4.slice/crio-11fd6a70a57211fff6a0b728b614fb8cc5322dd51c9946edae2fc95f0a7aedb9 WatchSource:0}: Error finding container 11fd6a70a57211fff6a0b728b614fb8cc5322dd51c9946edae2fc95f0a7aedb9: Status 404 returned error can't find the container with id 11fd6a70a57211fff6a0b728b614fb8cc5322dd51c9946edae2fc95f0a7aedb9
Oct 31 04:11:34.792028 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:34.791986    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm" event={"ID":"10f36a0c-b261-4862-b6fa-f8c3747fe6a4","Type":"ContainerStarted","Data":"5641149d1c737e586a4a988a0de174bb32e2b6bebb89e4baa6d4ec68b9be5f46"}
Oct 31 04:11:34.792437 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:34.792054    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm" event={"ID":"10f36a0c-b261-4862-b6fa-f8c3747fe6a4","Type":"ContainerStarted","Data":"11fd6a70a57211fff6a0b728b614fb8cc5322dd51c9946edae2fc95f0a7aedb9"}
Oct 31 04:11:34.792437 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:34.792087    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"
Oct 31 04:11:34.900115 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:34.900057    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm" podStartSLOduration=1.900043656 podStartE2EDuration="1.900043656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:34.899206364 +0000 UTC m=+3836.459686094" watchObservedRunningTime="2026-10-31 04:11:34.900043656 +0000 UTC m=+3836.460523386"
Oct 31 04:11:45.801013 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:45.800978    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"
Oct 31 04:11:46.176799 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:11:46.176763    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:11:56.359901 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.359841    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"]
Oct 31 04:11:56.363993 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.363966    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:56.413636 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.413605    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e53098-serving-cert\""
Oct 31 04:11:56.462744 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.462702    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"]
Oct 31 04:11:56.509990 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.509953    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-metrics-certs\") pod \"maas-api-e2e-inf-a-e53098-6fc456498b-d8msl\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:56.510191 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.510087    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e53098-6fc456498b-d8msl\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:56.510191 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.510134    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/782076e8-3b96-49a9-98f7-4134fc68c68b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e53098-6fc456498b-d8msl\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:56.510289 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.510271    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pldr4\" (UniqueName: \"kubernetes.io/projected/782076e8-3b96-49a9-98f7-4134fc68c68b-kube-api-access-pldr4\") pod \"maas-api-e2e-inf-a-e53098-6fc456498b-d8msl\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:56.611406 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.611314    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-metrics-certs\") pod \"maas-api-e2e-inf-a-e53098-6fc456498b-d8msl\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:56.611590 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.611415    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e53098-6fc456498b-d8msl\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:56.611590 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.611450    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/782076e8-3b96-49a9-98f7-4134fc68c68b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e53098-6fc456498b-d8msl\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:56.611590 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.611530    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pldr4\" (UniqueName: \"kubernetes.io/projected/782076e8-3b96-49a9-98f7-4134fc68c68b-kube-api-access-pldr4\") pod \"maas-api-e2e-inf-a-e53098-6fc456498b-d8msl\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:56.611590 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:11:56.611553    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-serving-cert: secret "maas-api-e2e-inf-a-e53098-serving-cert" not found
Oct 31 04:11:56.611827 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:11:56.611611    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-maas-api-tls podName:782076e8-3b96-49a9-98f7-4134fc68c68b nodeName:}" failed. No retries permitted until 2026-10-31 04:11:57.111595007 +0000 UTC m=+3858.672074724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-maas-api-tls") pod "maas-api-e2e-inf-a-e53098-6fc456498b-d8msl" (UID: "782076e8-3b96-49a9-98f7-4134fc68c68b") : secret "maas-api-e2e-inf-a-e53098-serving-cert" not found
Oct 31 04:11:56.612125 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.612106    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/782076e8-3b96-49a9-98f7-4134fc68c68b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e53098-6fc456498b-d8msl\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:56.613760 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.613738    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-metrics-certs\") pod \"maas-api-e2e-inf-a-e53098-6fc456498b-d8msl\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:56.668943 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:56.668911    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pldr4\" (UniqueName: \"kubernetes.io/projected/782076e8-3b96-49a9-98f7-4134fc68c68b-kube-api-access-pldr4\") pod \"maas-api-e2e-inf-a-e53098-6fc456498b-d8msl\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:57.117827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:57.117781    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e53098-6fc456498b-d8msl\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:57.120355 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:57.120326    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e53098-6fc456498b-d8msl\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:57.278772 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:57.278737    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:57.659097 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:57.659069    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"]
Oct 31 04:11:57.661190 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:11:57.661158    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782076e8_3b96_49a9_98f7_4134fc68c68b.slice/crio-a029d923ebdc729b6bbd31e608bd07f529128699496cfa571ec23222cec2f8a3 WatchSource:0}: Error finding container a029d923ebdc729b6bbd31e608bd07f529128699496cfa571ec23222cec2f8a3: Status 404 returned error can't find the container with id a029d923ebdc729b6bbd31e608bd07f529128699496cfa571ec23222cec2f8a3
Oct 31 04:11:57.662775 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:57.662758    5415 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:11:57.924254 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:57.924161    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl" event={"ID":"782076e8-3b96-49a9-98f7-4134fc68c68b","Type":"ContainerStarted","Data":"a029d923ebdc729b6bbd31e608bd07f529128699496cfa571ec23222cec2f8a3"}
Oct 31 04:11:58.931691 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:58.931581    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl" event={"ID":"782076e8-3b96-49a9-98f7-4134fc68c68b","Type":"ContainerStarted","Data":"352253814d226c54050a23c994bf59feffc86e86c8612090b327ba3c2704edb6"}
Oct 31 04:11:58.932146 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:58.931696    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:11:59.040949 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:11:59.040843    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl" podStartSLOduration=2.104483243 podStartE2EDuration="3.040824715s" totalImagesPullingTime="936.341472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:59.039539662 +0000 UTC m=+3860.600019394" watchObservedRunningTime="2026-10-31 04:11:59.040824715 +0000 UTC m=+3860.601304449"
Oct 31 04:12:04.944943 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:04.944911    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:12:16.177030 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:12:16.176943    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:12:46.176055 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:12:46.176015    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:12:48.296918 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:48.296869    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"]
Oct 31 04:12:48.301442 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:48.301411    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"
Oct 31 04:12:48.399322 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:48.399277    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-243a29-dockercfg-9qlp8\""
Oct 31 04:12:48.399322 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:48.399277    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-243a29\""
Oct 31 04:12:48.401890 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:48.401842    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"]
Oct 31 04:12:48.431342 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:48.431311    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lglg2\" (UniqueName: \"kubernetes.io/projected/76bfbd25-939a-48b1-ae8e-e3cbf1313530-kube-api-access-lglg2\") pod \"payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw\" (UID: \"76bfbd25-939a-48b1-ae8e-e3cbf1313530\") " pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"
Oct 31 04:12:48.431544 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:48.431520    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bfbd25-939a-48b1-ae8e-e3cbf1313530-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw\" (UID: \"76bfbd25-939a-48b1-ae8e-e3cbf1313530\") " pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"
Oct 31 04:12:48.533025 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:48.532992    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bfbd25-939a-48b1-ae8e-e3cbf1313530-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw\" (UID: \"76bfbd25-939a-48b1-ae8e-e3cbf1313530\") " pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"
Oct 31 04:12:48.533250 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:48.533113    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lglg2\" (UniqueName: \"kubernetes.io/projected/76bfbd25-939a-48b1-ae8e-e3cbf1313530-kube-api-access-lglg2\") pod \"payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw\" (UID: \"76bfbd25-939a-48b1-ae8e-e3cbf1313530\") " pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"
Oct 31 04:12:48.533790 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:48.533770    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bfbd25-939a-48b1-ae8e-e3cbf1313530-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw\" (UID: \"76bfbd25-939a-48b1-ae8e-e3cbf1313530\") " pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"
Oct 31 04:12:48.587079 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:48.587040    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lglg2\" (UniqueName: \"kubernetes.io/projected/76bfbd25-939a-48b1-ae8e-e3cbf1313530-kube-api-access-lglg2\") pod \"payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw\" (UID: \"76bfbd25-939a-48b1-ae8e-e3cbf1313530\") " pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"
Oct 31 04:12:48.621937 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:48.621869    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"
Oct 31 04:12:48.990372 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:48.990338    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"]
Oct 31 04:12:48.992791 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:12:48.992742    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bfbd25_939a_48b1_ae8e_e3cbf1313530.slice/crio-f2a2bcac0fd4b5766b77fe66cad0395a31273463c9fda8d0bec94f6b96ca067d WatchSource:0}: Error finding container f2a2bcac0fd4b5766b77fe66cad0395a31273463c9fda8d0bec94f6b96ca067d: Status 404 returned error can't find the container with id f2a2bcac0fd4b5766b77fe66cad0395a31273463c9fda8d0bec94f6b96ca067d
Oct 31 04:12:49.224199 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:49.224160    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw" event={"ID":"76bfbd25-939a-48b1-ae8e-e3cbf1313530","Type":"ContainerStarted","Data":"43a68416f66811dbd1dbff15c737f036b70b5c8f80efeb52d51666a92becd7aa"}
Oct 31 04:12:49.224397 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:49.224229    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw" event={"ID":"76bfbd25-939a-48b1-ae8e-e3cbf1313530","Type":"ContainerStarted","Data":"f2a2bcac0fd4b5766b77fe66cad0395a31273463c9fda8d0bec94f6b96ca067d"}
Oct 31 04:12:49.224397 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:49.224373    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"
Oct 31 04:12:49.332569 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:49.332486    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw" podStartSLOduration=1.332468671 podStartE2EDuration="1.332468671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:49.33077868 +0000 UTC m=+3910.891258411" watchObservedRunningTime="2026-10-31 04:12:49.332468671 +0000 UTC m=+3910.892948419"
Oct 31 04:12:52.429364 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.429321    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"]
Oct 31 04:12:52.433477 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.433450    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:52.482768 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.482730    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e2a54d-serving-cert\""
Oct 31 04:12:52.532279 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.532242    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"]
Oct 31 04:12:52.586795 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.586749    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s94n\" (UniqueName: \"kubernetes.io/projected/5bfa71e4-f250-4228-8e7d-426f43abffd5-kube-api-access-5s94n\") pod \"maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:52.587028 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.586957    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-metrics-certs\") pod \"maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:52.587110 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.587074    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfa71e4-f250-4228-8e7d-426f43abffd5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:52.587174 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.587134    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:52.688567 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.688481    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-metrics-certs\") pod \"maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:52.688567 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.688557    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfa71e4-f250-4228-8e7d-426f43abffd5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:52.688811 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.688593    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:52.688811 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.688679    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s94n\" (UniqueName: \"kubernetes.io/projected/5bfa71e4-f250-4228-8e7d-426f43abffd5-kube-api-access-5s94n\") pod \"maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:52.688811 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:12:52.688799    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-serving-cert: secret "maas-api-e2e-inf-a-e2a54d-serving-cert" not found
Oct 31 04:12:52.689029 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:12:52.688914    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-maas-api-tls podName:5bfa71e4-f250-4228-8e7d-426f43abffd5 nodeName:}" failed. No retries permitted until 2026-10-31 04:12:53.188890008 +0000 UTC m=+3914.749369736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-maas-api-tls") pod "maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds" (UID: "5bfa71e4-f250-4228-8e7d-426f43abffd5") : secret "maas-api-e2e-inf-a-e2a54d-serving-cert" not found
Oct 31 04:12:52.689406 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.689376    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfa71e4-f250-4228-8e7d-426f43abffd5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:52.691633 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.691606    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-metrics-certs\") pod \"maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:52.747536 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:52.747505    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s94n\" (UniqueName: \"kubernetes.io/projected/5bfa71e4-f250-4228-8e7d-426f43abffd5-kube-api-access-5s94n\") pod \"maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:53.195627 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:53.195576    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:53.198230 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:53.198207    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:53.347546 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:53.347511    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:53.717101 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:53.717072    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"]
Oct 31 04:12:53.719333 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:12:53.719296    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfa71e4_f250_4228_8e7d_426f43abffd5.slice/crio-5fb6a23a218fd7c7ce10620028e5712c1739bcdfbcc83952e5416d5dea16a025 WatchSource:0}: Error finding container 5fb6a23a218fd7c7ce10620028e5712c1739bcdfbcc83952e5416d5dea16a025: Status 404 returned error can't find the container with id 5fb6a23a218fd7c7ce10620028e5712c1739bcdfbcc83952e5416d5dea16a025
Oct 31 04:12:54.253507 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:54.253467    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds" event={"ID":"5bfa71e4-f250-4228-8e7d-426f43abffd5","Type":"ContainerStarted","Data":"5fb6a23a218fd7c7ce10620028e5712c1739bcdfbcc83952e5416d5dea16a025"}
Oct 31 04:12:55.261559 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:55.261523    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds" event={"ID":"5bfa71e4-f250-4228-8e7d-426f43abffd5","Type":"ContainerStarted","Data":"1d8f83200b1552be3a85fbc25632b655c9134bd3de2ecd99c4ec70b75d0a2c17"}
Oct 31 04:12:55.262053 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:55.261610    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:12:55.372258 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:12:55.372203    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds" podStartSLOduration=2.4599585409999998 podStartE2EDuration="3.372183863s" totalImagesPullingTime="912.225322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:55.371268369 +0000 UTC m=+3916.931748101" watchObservedRunningTime="2026-10-31 04:12:55.372183863 +0000 UTC m=+3916.932663708"
Oct 31 04:13:00.233137 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:00.233103    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"
Oct 31 04:13:01.274532 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:01.274500    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:13:16.176807 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:13:16.176770    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:13:44.009101 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:44.009063    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"]
Oct 31 04:13:44.013013 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:44.012987    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"
Oct 31 04:13:44.062685 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:44.062651    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e679fe\""
Oct 31 04:13:44.109739 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:44.109683    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e679fe-dockercfg-49tzp\""
Oct 31 04:13:44.110345 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:44.110307    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t\" (UID: \"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"
Oct 31 04:13:44.110580 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:44.110557    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrmk\" (UniqueName: \"kubernetes.io/projected/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-kube-api-access-lrrmk\") pod \"payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t\" (UID: \"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"
Oct 31 04:13:44.112408 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:44.112381    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"]
Oct 31 04:13:44.212509 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:44.212439    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t\" (UID: \"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"
Oct 31 04:13:44.212724 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:44.212527    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrmk\" (UniqueName: \"kubernetes.io/projected/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-kube-api-access-lrrmk\") pod \"payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t\" (UID: \"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"
Oct 31 04:13:44.213312 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:44.213266    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t\" (UID: \"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"
Oct 31 04:13:44.268393 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:44.268311    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrmk\" (UniqueName: \"kubernetes.io/projected/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-kube-api-access-lrrmk\") pod \"payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t\" (UID: \"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"
Oct 31 04:13:44.327422 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:44.327375    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"
Oct 31 04:13:44.696547 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:44.696518    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"]
Oct 31 04:13:44.698840 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:13:44.698809    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321cdce7_96ef_4f96_83b3_2f6a2ba1c6d2.slice/crio-501001b9ee262519f795f1130634d38cacfa0929a968651c1effd6403e3272e3 WatchSource:0}: Error finding container 501001b9ee262519f795f1130634d38cacfa0929a968651c1effd6403e3272e3: Status 404 returned error can't find the container with id 501001b9ee262519f795f1130634d38cacfa0929a968651c1effd6403e3272e3
Oct 31 04:13:45.572296 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:45.572079    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t" event={"ID":"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2","Type":"ContainerStarted","Data":"e603f5f8cf3b2684c1b2f1cb2065c551edbdb2573834e87153d89d74258bea77"}
Oct 31 04:13:45.572296 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:45.572152    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t" event={"ID":"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2","Type":"ContainerStarted","Data":"501001b9ee262519f795f1130634d38cacfa0929a968651c1effd6403e3272e3"}
Oct 31 04:13:45.572296 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:45.572183    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"
Oct 31 04:13:45.681391 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:45.681334    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t" podStartSLOduration=2.68131985 podStartE2EDuration="2.68131985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:45.679596925 +0000 UTC m=+3967.240076675" watchObservedRunningTime="2026-10-31 04:13:45.68131985 +0000 UTC m=+3967.241799634"
Oct 31 04:13:46.175818 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:13:46.175782    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:13:56.581719 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:13:56.581688    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"
Oct 31 04:14:09.516641 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:09.516599    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"]
Oct 31 04:14:14.519249 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:14.519195    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss" podUID="a8b3f000-ea82-4cb7-9d3a-8582790da783" containerName="payload-pre-processing" containerID="cri-o://df3e1782cbf4de3e2df83a20eb74fe37f24a02636f807d7ccd78a318157791bf" gracePeriod=25
Oct 31 04:14:14.747816 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:14.747782    5415 generic.go:356] "Generic (PLEG): container finished" podID="a8b3f000-ea82-4cb7-9d3a-8582790da783" containerID="df3e1782cbf4de3e2df83a20eb74fe37f24a02636f807d7ccd78a318157791bf" exitCode=0
Oct 31 04:14:14.748041 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:14.747868    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss" event={"ID":"a8b3f000-ea82-4cb7-9d3a-8582790da783","Type":"ContainerDied","Data":"df3e1782cbf4de3e2df83a20eb74fe37f24a02636f807d7ccd78a318157791bf"}
Oct 31 04:14:14.808709 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:14.808680    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"
Oct 31 04:14:14.887277 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:14.887238    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b3f000-ea82-4cb7-9d3a-8582790da783-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b3f000-ea82-4cb7-9d3a-8582790da783-plugins-config-volume\") pod \"a8b3f000-ea82-4cb7-9d3a-8582790da783\" (UID: \"a8b3f000-ea82-4cb7-9d3a-8582790da783\") "
Oct 31 04:14:14.887465 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:14.887316    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b3f000-ea82-4cb7-9d3a-8582790da783-kube-api-access-97gr5\" (UniqueName: \"kubernetes.io/projected/a8b3f000-ea82-4cb7-9d3a-8582790da783-kube-api-access-97gr5\") pod \"a8b3f000-ea82-4cb7-9d3a-8582790da783\" (UID: \"a8b3f000-ea82-4cb7-9d3a-8582790da783\") "
Oct 31 04:14:14.887649 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:14.887629    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b3f000-ea82-4cb7-9d3a-8582790da783-plugins-config-volume" pod "a8b3f000-ea82-4cb7-9d3a-8582790da783" (UID: "a8b3f000-ea82-4cb7-9d3a-8582790da783"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:14:14.889444 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:14.889415    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b3f000-ea82-4cb7-9d3a-8582790da783-kube-api-access-97gr5" pod "a8b3f000-ea82-4cb7-9d3a-8582790da783" (UID: "a8b3f000-ea82-4cb7-9d3a-8582790da783"). InnerVolumeSpecName "kube-api-access-97gr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:14:14.988704 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:14.988665    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b3f000-ea82-4cb7-9d3a-8582790da783-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:14.988704 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:14.988700    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97gr5\" (UniqueName: \"kubernetes.io/projected/a8b3f000-ea82-4cb7-9d3a-8582790da783-kube-api-access-97gr5\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:15.156728 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.156691    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"]
Oct 31 04:14:15.157317 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.157301    5415 cpu_manager.go:414] "containerMap: removing container" podUID="a8b3f000-ea82-4cb7-9d3a-8582790da783" containerName="payload-pre-processing"
Oct 31 04:14:15.157317 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.157319    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b3f000-ea82-4cb7-9d3a-8582790da783" containerName="payload-pre-processing"
Oct 31 04:14:15.157418 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.157407    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b3f000-ea82-4cb7-9d3a-8582790da783" containerName="payload-pre-processing"
Oct 31 04:14:15.160836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.160814    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.209949 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.209912    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-243a29-serving-cert\""
Oct 31 04:14:15.218504 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.218470    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"]
Oct 31 04:14:15.292381 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.292303    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af401f3-6429-4711-bcaf-56c6bba19998-openshift-service-ca\") pod \"maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.292612 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.292486    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q2wx\" (UniqueName: \"kubernetes.io/projected/0af401f3-6429-4711-bcaf-56c6bba19998-kube-api-access-5q2wx\") pod \"maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.292733 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.292712    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-maas-api-tls\") pod \"maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.292796 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.292750    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-metrics-certs\") pod \"maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.394104 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.394052    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-maas-api-tls\") pod \"maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.394104 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.394102    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-metrics-certs\") pod \"maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.394413 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.394178    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af401f3-6429-4711-bcaf-56c6bba19998-openshift-service-ca\") pod \"maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.394413 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.394238    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q2wx\" (UniqueName: \"kubernetes.io/projected/0af401f3-6429-4711-bcaf-56c6bba19998-kube-api-access-5q2wx\") pod \"maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.394413 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:14:15.394245    5415 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-serving-cert: secret "maas-api-e2e-inf-b-243a29-serving-cert" not found
Oct 31 04:14:15.394413 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:14:15.394345    5415 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-maas-api-tls podName:0af401f3-6429-4711-bcaf-56c6bba19998 nodeName:}" failed. No retries permitted until 2026-10-31 04:14:15.894318622 +0000 UTC m=+3997.454798334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-maas-api-tls") pod "maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr" (UID: "0af401f3-6429-4711-bcaf-56c6bba19998") : secret "maas-api-e2e-inf-b-243a29-serving-cert" not found
Oct 31 04:14:15.395103 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.395084    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af401f3-6429-4711-bcaf-56c6bba19998-openshift-service-ca\") pod \"maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.396740 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.396716    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-metrics-certs\") pod \"maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.448805 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.448728    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q2wx\" (UniqueName: \"kubernetes.io/projected/0af401f3-6429-4711-bcaf-56c6bba19998-kube-api-access-5q2wx\") pod \"maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.754777 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.754691    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"
Oct 31 04:14:15.755201 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.754687    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss" event={"ID":"a8b3f000-ea82-4cb7-9d3a-8582790da783","Type":"ContainerDied","Data":"bb834d30d64c676c0c84634bc863b659835f6f1c11a53ba648e6f3d0d285f74a"}
Oct 31 04:14:15.755201 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.754844    5415 scope.go:122] "RemoveContainer" containerID="df3e1782cbf4de3e2df83a20eb74fe37f24a02636f807d7ccd78a318157791bf"
Oct 31 04:14:15.899689 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.899640    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-maas-api-tls\") pod \"maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.902278 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.902254    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-maas-api-tls\") pod \"maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:15.909421 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.909382    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"]
Oct 31 04:14:15.912720 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:15.912690    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d247af-5c87c68687-fdkss"]
Oct 31 04:14:16.074996 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:16.074859    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:16.182106 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:14:16.182063    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:14:16.496667 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:16.496636    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"]
Oct 31 04:14:16.498749 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:14:16.498702    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af401f3_6429_4711_bcaf_56c6bba19998.slice/crio-3fde2edf4f7fb3a13e5167b1ae007af85c7e896920a77d2cf1dee0c6a81af90e WatchSource:0}: Error finding container 3fde2edf4f7fb3a13e5167b1ae007af85c7e896920a77d2cf1dee0c6a81af90e: Status 404 returned error can't find the container with id 3fde2edf4f7fb3a13e5167b1ae007af85c7e896920a77d2cf1dee0c6a81af90e
Oct 31 04:14:16.606443 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:16.606410    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b3f000-ea82-4cb7-9d3a-8582790da783" path="/var/lib/kubelet/pods/a8b3f000-ea82-4cb7-9d3a-8582790da783/volumes"
Oct 31 04:14:16.762935 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:16.762833    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr" event={"ID":"0af401f3-6429-4711-bcaf-56c6bba19998","Type":"ContainerStarted","Data":"3fde2edf4f7fb3a13e5167b1ae007af85c7e896920a77d2cf1dee0c6a81af90e"}
Oct 31 04:14:17.770090 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:17.769996    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr" event={"ID":"0af401f3-6429-4711-bcaf-56c6bba19998","Type":"ContainerStarted","Data":"e8f36e6145a67e44cddbf2a3144b921adcbdd3a2189e6d0c1b73ae903575ae3f"}
Oct 31 04:14:17.770090 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:17.770079    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:17.878505 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:17.878452    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr" podStartSLOduration=1.939199422 podStartE2EDuration="2.878437194s" totalImagesPullingTime="939.237772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:17.878084819 +0000 UTC m=+3999.438564571" watchObservedRunningTime="2026-10-31 04:14:17.878437194 +0000 UTC m=+3999.438917039"
Oct 31 04:14:23.781836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:23.781803    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:14:34.889246 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:34.889211    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"]
Oct 31 04:14:39.890315 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:39.890209    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm" podUID="10f36a0c-b261-4862-b6fa-f8c3747fe6a4" containerName="payload-pre-processing" containerID="cri-o://5641149d1c737e586a4a988a0de174bb32e2b6bebb89e4baa6d4ec68b9be5f46" gracePeriod=25
Oct 31 04:14:40.162892 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.162850    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"
Oct 31 04:14:40.290492 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.290457    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-kube-api-access-jvtxb\" (UniqueName: \"kubernetes.io/projected/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-kube-api-access-jvtxb\") pod \"10f36a0c-b261-4862-b6fa-f8c3747fe6a4\" (UID: \"10f36a0c-b261-4862-b6fa-f8c3747fe6a4\") "
Oct 31 04:14:40.290713 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.290691    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-plugins-config-volume\") pod \"10f36a0c-b261-4862-b6fa-f8c3747fe6a4\" (UID: \"10f36a0c-b261-4862-b6fa-f8c3747fe6a4\") "
Oct 31 04:14:40.291074 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.291049    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-plugins-config-volume" pod "10f36a0c-b261-4862-b6fa-f8c3747fe6a4" (UID: "10f36a0c-b261-4862-b6fa-f8c3747fe6a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:14:40.292675 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.292653    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-kube-api-access-jvtxb" pod "10f36a0c-b261-4862-b6fa-f8c3747fe6a4" (UID: "10f36a0c-b261-4862-b6fa-f8c3747fe6a4"). InnerVolumeSpecName "kube-api-access-jvtxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:14:40.392841 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.392753    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvtxb\" (UniqueName: \"kubernetes.io/projected/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-kube-api-access-jvtxb\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:40.392841 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.392785    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f36a0c-b261-4862-b6fa-f8c3747fe6a4-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:40.911815 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.911779    5415 generic.go:356] "Generic (PLEG): container finished" podID="10f36a0c-b261-4862-b6fa-f8c3747fe6a4" containerID="5641149d1c737e586a4a988a0de174bb32e2b6bebb89e4baa6d4ec68b9be5f46" exitCode=0
Oct 31 04:14:40.912260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.911828    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"
Oct 31 04:14:40.912260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.911868    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm" event={"ID":"10f36a0c-b261-4862-b6fa-f8c3747fe6a4","Type":"ContainerDied","Data":"5641149d1c737e586a4a988a0de174bb32e2b6bebb89e4baa6d4ec68b9be5f46"}
Oct 31 04:14:40.912260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.911951    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm" event={"ID":"10f36a0c-b261-4862-b6fa-f8c3747fe6a4","Type":"ContainerDied","Data":"11fd6a70a57211fff6a0b728b614fb8cc5322dd51c9946edae2fc95f0a7aedb9"}
Oct 31 04:14:40.912260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.911976    5415 scope.go:122] "RemoveContainer" containerID="5641149d1c737e586a4a988a0de174bb32e2b6bebb89e4baa6d4ec68b9be5f46"
Oct 31 04:14:40.922331 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.922307    5415 scope.go:122] "RemoveContainer" containerID="5641149d1c737e586a4a988a0de174bb32e2b6bebb89e4baa6d4ec68b9be5f46"
Oct 31 04:14:40.922672 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:14:40.922652    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5641149d1c737e586a4a988a0de174bb32e2b6bebb89e4baa6d4ec68b9be5f46\": container with ID starting with 5641149d1c737e586a4a988a0de174bb32e2b6bebb89e4baa6d4ec68b9be5f46 not found: ID does not exist" containerID="5641149d1c737e586a4a988a0de174bb32e2b6bebb89e4baa6d4ec68b9be5f46"
Oct 31 04:14:40.922744 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:40.922683    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5641149d1c737e586a4a988a0de174bb32e2b6bebb89e4baa6d4ec68b9be5f46"} err="failed to get container status \"5641149d1c737e586a4a988a0de174bb32e2b6bebb89e4baa6d4ec68b9be5f46\": rpc error: code = NotFound desc = could not find container \"5641149d1c737e586a4a988a0de174bb32e2b6bebb89e4baa6d4ec68b9be5f46\": container with ID starting with 5641149d1c737e586a4a988a0de174bb32e2b6bebb89e4baa6d4ec68b9be5f46 not found: ID does not exist"
Oct 31 04:14:41.066367 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:41.066332    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"]
Oct 31 04:14:41.113038 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:41.112995    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e2a54d-6d5467ddcf-942wm"]
Oct 31 04:14:42.606479 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:14:42.606441    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f36a0c-b261-4862-b6fa-f8c3747fe6a4" path="/var/lib/kubelet/pods/10f36a0c-b261-4862-b6fa-f8c3747fe6a4/volumes"
Oct 31 04:14:46.176026 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:14:46.175992    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:15:00.192100 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.192060    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"]
Oct 31 04:15:00.192846 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.192822    5415 cpu_manager.go:414] "containerMap: removing container" podUID="10f36a0c-b261-4862-b6fa-f8c3747fe6a4" containerName="payload-pre-processing"
Oct 31 04:15:00.192977 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.192847    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f36a0c-b261-4862-b6fa-f8c3747fe6a4" containerName="payload-pre-processing"
Oct 31 04:15:00.193036 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.192996    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f36a0c-b261-4862-b6fa-f8c3747fe6a4" containerName="payload-pre-processing"
Oct 31 04:15:00.196848 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.196821    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"]
Oct 31 04:15:00.196986 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.196967    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"
Oct 31 04:15:00.200585 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.200560    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"
Oct 31 04:15:00.214329 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.214304    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"]
Oct 31 04:15:00.218264 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.218239    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"
Oct 31 04:15:00.234814 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.234785    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt7fb\" (UniqueName: \"kubernetes.io/projected/5e4f1595-2c8a-4408-a037-9de9b29669cb-kube-api-access-jt7fb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b\" (UID: \"5e4f1595-2c8a-4408-a037-9de9b29669cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"
Oct 31 04:15:00.234994 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.234832    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pcls\" (UniqueName: \"kubernetes.io/projected/b13e4466-9375-4740-9ba7-12d286d0142a-kube-api-access-4pcls\") pod \"maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l\" (UID: \"b13e4466-9375-4740-9ba7-12d286d0142a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"
Oct 31 04:15:00.234994 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.234973    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13e4466-9375-4740-9ba7-12d286d0142a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l\" (UID: \"b13e4466-9375-4740-9ba7-12d286d0142a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"
Oct 31 04:15:00.235091 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.235032    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f1595-2c8a-4408-a037-9de9b29669cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b\" (UID: \"5e4f1595-2c8a-4408-a037-9de9b29669cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"
Oct 31 04:15:00.239614 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.239583    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"]
Oct 31 04:15:00.243670 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.243650    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"]
Oct 31 04:15:00.243810 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.243789    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"
Oct 31 04:15:00.247523 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.247498    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"
Oct 31 04:15:00.333235 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.333206    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:15:00.336839 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.336797    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc5z2\" (UniqueName: \"kubernetes.io/projected/d8034d48-df5b-40ae-a7a0-c026aad438ed-kube-api-access-jc5z2\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq\" (UID: \"d8034d48-df5b-40ae-a7a0-c026aad438ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"
Oct 31 04:15:00.337002 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.336951    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8034d48-df5b-40ae-a7a0-c026aad438ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq\" (UID: \"d8034d48-df5b-40ae-a7a0-c026aad438ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"
Oct 31 04:15:00.337049 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.337009    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt7fb\" (UniqueName: \"kubernetes.io/projected/5e4f1595-2c8a-4408-a037-9de9b29669cb-kube-api-access-jt7fb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b\" (UID: \"5e4f1595-2c8a-4408-a037-9de9b29669cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"
Oct 31 04:15:00.337049 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.337028    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa92af3-a192-4c80-bdb0-097b66fe9f14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c\" (UID: \"6fa92af3-a192-4c80-bdb0-097b66fe9f14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"
Oct 31 04:15:00.337257 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.337208    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pcls\" (UniqueName: \"kubernetes.io/projected/b13e4466-9375-4740-9ba7-12d286d0142a-kube-api-access-4pcls\") pod \"maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l\" (UID: \"b13e4466-9375-4740-9ba7-12d286d0142a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"
Oct 31 04:15:00.337316 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.337295    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13e4466-9375-4740-9ba7-12d286d0142a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l\" (UID: \"b13e4466-9375-4740-9ba7-12d286d0142a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"
Oct 31 04:15:00.337407 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.337378    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a70215e-fc06-4236-91cb-bda11d72454b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr\" (UID: \"6a70215e-fc06-4236-91cb-bda11d72454b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"
Oct 31 04:15:00.337470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.337420    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbwf7\" (UniqueName: \"kubernetes.io/projected/6fa92af3-a192-4c80-bdb0-097b66fe9f14-kube-api-access-nbwf7\") pod \"maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c\" (UID: \"6fa92af3-a192-4c80-bdb0-097b66fe9f14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"
Oct 31 04:15:00.337470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.337438    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlmxs\" (UniqueName: \"kubernetes.io/projected/6a70215e-fc06-4236-91cb-bda11d72454b-kube-api-access-hlmxs\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr\" (UID: \"6a70215e-fc06-4236-91cb-bda11d72454b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"
Oct 31 04:15:00.337578 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.337474    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f1595-2c8a-4408-a037-9de9b29669cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b\" (UID: \"5e4f1595-2c8a-4408-a037-9de9b29669cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"
Oct 31 04:15:00.338059 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.338039    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13e4466-9375-4740-9ba7-12d286d0142a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l\" (UID: \"b13e4466-9375-4740-9ba7-12d286d0142a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"
Oct 31 04:15:00.338106 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.338091    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f1595-2c8a-4408-a037-9de9b29669cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b\" (UID: \"5e4f1595-2c8a-4408-a037-9de9b29669cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"
Oct 31 04:15:00.385816 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.385781    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"]
Oct 31 04:15:00.388527 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.388109    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"]
Oct 31 04:15:00.390984 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.390947    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"]
Oct 31 04:15:00.400749 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.400716    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pcls\" (UniqueName: \"kubernetes.io/projected/b13e4466-9375-4740-9ba7-12d286d0142a-kube-api-access-4pcls\") pod \"maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l\" (UID: \"b13e4466-9375-4740-9ba7-12d286d0142a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"
Oct 31 04:15:00.402448 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.402417    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"]
Oct 31 04:15:00.403089 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.403061    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt7fb\" (UniqueName: \"kubernetes.io/projected/5e4f1595-2c8a-4408-a037-9de9b29669cb-kube-api-access-jt7fb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b\" (UID: \"5e4f1595-2c8a-4408-a037-9de9b29669cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"
Oct 31 04:15:00.439357 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.439312    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa92af3-a192-4c80-bdb0-097b66fe9f14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c\" (UID: \"6fa92af3-a192-4c80-bdb0-097b66fe9f14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"
Oct 31 04:15:00.439539 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.439501    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a70215e-fc06-4236-91cb-bda11d72454b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr\" (UID: \"6a70215e-fc06-4236-91cb-bda11d72454b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"
Oct 31 04:15:00.439539 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.439531    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbwf7\" (UniqueName: \"kubernetes.io/projected/6fa92af3-a192-4c80-bdb0-097b66fe9f14-kube-api-access-nbwf7\") pod \"maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c\" (UID: \"6fa92af3-a192-4c80-bdb0-097b66fe9f14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"
Oct 31 04:15:00.439617 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.439560    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlmxs\" (UniqueName: \"kubernetes.io/projected/6a70215e-fc06-4236-91cb-bda11d72454b-kube-api-access-hlmxs\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr\" (UID: \"6a70215e-fc06-4236-91cb-bda11d72454b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"
Oct 31 04:15:00.439663 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.439635    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc5z2\" (UniqueName: \"kubernetes.io/projected/d8034d48-df5b-40ae-a7a0-c026aad438ed-kube-api-access-jc5z2\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq\" (UID: \"d8034d48-df5b-40ae-a7a0-c026aad438ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"
Oct 31 04:15:00.439742 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.439725    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8034d48-df5b-40ae-a7a0-c026aad438ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq\" (UID: \"d8034d48-df5b-40ae-a7a0-c026aad438ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"
Oct 31 04:15:00.440181 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.440157    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa92af3-a192-4c80-bdb0-097b66fe9f14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c\" (UID: \"6fa92af3-a192-4c80-bdb0-097b66fe9f14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"
Oct 31 04:15:00.440252 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.440231    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a70215e-fc06-4236-91cb-bda11d72454b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr\" (UID: \"6a70215e-fc06-4236-91cb-bda11d72454b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"
Oct 31 04:15:00.440566 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.440543    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8034d48-df5b-40ae-a7a0-c026aad438ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq\" (UID: \"d8034d48-df5b-40ae-a7a0-c026aad438ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"
Oct 31 04:15:00.455341 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.455310    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"]
Oct 31 04:15:00.494461 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.494409    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbwf7\" (UniqueName: \"kubernetes.io/projected/6fa92af3-a192-4c80-bdb0-097b66fe9f14-kube-api-access-nbwf7\") pod \"maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c\" (UID: \"6fa92af3-a192-4c80-bdb0-097b66fe9f14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"
Oct 31 04:15:00.494676 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.494588    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc5z2\" (UniqueName: \"kubernetes.io/projected/d8034d48-df5b-40ae-a7a0-c026aad438ed-kube-api-access-jc5z2\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq\" (UID: \"d8034d48-df5b-40ae-a7a0-c026aad438ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"
Oct 31 04:15:00.495308 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.495283    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlmxs\" (UniqueName: \"kubernetes.io/projected/6a70215e-fc06-4236-91cb-bda11d72454b-kube-api-access-hlmxs\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr\" (UID: \"6a70215e-fc06-4236-91cb-bda11d72454b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"
Oct 31 04:15:00.515508 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.515474    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"
Oct 31 04:15:00.519743 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.519712    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"
Oct 31 04:15:00.532084 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.532049    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"
Oct 31 04:15:00.559720 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.559621    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"
Oct 31 04:15:00.572940 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.568183    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"
Oct 31 04:15:00.897237 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.897210    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"]
Oct 31 04:15:00.899856 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:15:00.899823    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13e4466_9375_4740_9ba7_12d286d0142a.slice/crio-a7cd3996280308cfb8eeac1ef75d44b7e5c9020da52e4f3fcf09a4930853a23d WatchSource:0}: Error finding container a7cd3996280308cfb8eeac1ef75d44b7e5c9020da52e4f3fcf09a4930853a23d: Status 404 returned error can't find the container with id a7cd3996280308cfb8eeac1ef75d44b7e5c9020da52e4f3fcf09a4930853a23d
Oct 31 04:15:00.945689 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.945649    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"]
Oct 31 04:15:00.947521 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:15:00.947494    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a70215e_fc06_4236_91cb_bda11d72454b.slice/crio-b29d0f7901395a2ba4dbead9699b62e9f98a6c5c9da194b789eb84d72302bd19 WatchSource:0}: Error finding container b29d0f7901395a2ba4dbead9699b62e9f98a6c5c9da194b789eb84d72302bd19: Status 404 returned error can't find the container with id b29d0f7901395a2ba4dbead9699b62e9f98a6c5c9da194b789eb84d72302bd19
Oct 31 04:15:00.949159 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.949135    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"]
Oct 31 04:15:00.950201 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:15:00.950163    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4f1595_2c8a_4408_a037_9de9b29669cb.slice/crio-7cc4a6ff0ee94b2f6252295eaedb51da3e55b18f92788a8b6de33fd76cd46866 WatchSource:0}: Error finding container 7cc4a6ff0ee94b2f6252295eaedb51da3e55b18f92788a8b6de33fd76cd46866: Status 404 returned error can't find the container with id 7cc4a6ff0ee94b2f6252295eaedb51da3e55b18f92788a8b6de33fd76cd46866
Oct 31 04:15:00.995696 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:00.995671    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"]
Oct 31 04:15:01.016997 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:01.016964    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"]
Oct 31 04:15:01.045281 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:01.045246    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr" event={"ID":"6a70215e-fc06-4236-91cb-bda11d72454b","Type":"ContainerStarted","Data":"b29d0f7901395a2ba4dbead9699b62e9f98a6c5c9da194b789eb84d72302bd19"}
Oct 31 04:15:01.046601 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:01.046574    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l" event={"ID":"b13e4466-9375-4740-9ba7-12d286d0142a","Type":"ContainerStarted","Data":"a7cd3996280308cfb8eeac1ef75d44b7e5c9020da52e4f3fcf09a4930853a23d"}
Oct 31 04:15:01.048084 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:01.048049    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b" event={"ID":"5e4f1595-2c8a-4408-a037-9de9b29669cb","Type":"ContainerStarted","Data":"7cc4a6ff0ee94b2f6252295eaedb51da3e55b18f92788a8b6de33fd76cd46866"}
Oct 31 04:15:01.050459 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:15:01.050230    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8034d48_df5b_40ae_a7a0_c026aad438ed.slice/crio-c85ac2ccdcb6dbbbeb074ba266c81fb42f0bd14a42a6d65bd8e2b359478d8bbc WatchSource:0}: Error finding container c85ac2ccdcb6dbbbeb074ba266c81fb42f0bd14a42a6d65bd8e2b359478d8bbc: Status 404 returned error can't find the container with id c85ac2ccdcb6dbbbeb074ba266c81fb42f0bd14a42a6d65bd8e2b359478d8bbc
Oct 31 04:15:01.051439 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:15:01.051411    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa92af3_a192_4c80_bdb0_097b66fe9f14.slice/crio-754401d3e4f5328a215fc79e4ed53599249592e34c05279810fdfd8860f1abda WatchSource:0}: Error finding container 754401d3e4f5328a215fc79e4ed53599249592e34c05279810fdfd8860f1abda: Status 404 returned error can't find the container with id 754401d3e4f5328a215fc79e4ed53599249592e34c05279810fdfd8860f1abda
Oct 31 04:15:02.056742 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.056708    5415 generic.go:356] "Generic (PLEG): container finished" podID="5e4f1595-2c8a-4408-a037-9de9b29669cb" containerID="8ca0f173608a7c72cebc33d60a877a501f81888be30dedc570bf0b87e8c3acaf" exitCode=60
Oct 31 04:15:02.057173 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.056802    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b" event={"ID":"5e4f1595-2c8a-4408-a037-9de9b29669cb","Type":"ContainerDied","Data":"8ca0f173608a7c72cebc33d60a877a501f81888be30dedc570bf0b87e8c3acaf"}
Oct 31 04:15:02.057294 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.057269    5415 scope.go:122] "RemoveContainer" containerID="8ca0f173608a7c72cebc33d60a877a501f81888be30dedc570bf0b87e8c3acaf"
Oct 31 04:15:02.059253 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.059186    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr" event={"ID":"6a70215e-fc06-4236-91cb-bda11d72454b","Type":"ContainerStarted","Data":"ee04d61e5432bc390ed9ccf7604b1169d3c2fe5dbf73d4b953f46d97bd694ceb"}
Oct 31 04:15:02.059546 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.059530    5415 scope.go:122] "RemoveContainer" containerID="ee04d61e5432bc390ed9ccf7604b1169d3c2fe5dbf73d4b953f46d97bd694ceb"
Oct 31 04:15:02.065221 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.065190    5415 generic.go:356] "Generic (PLEG): container finished" podID="b13e4466-9375-4740-9ba7-12d286d0142a" containerID="f472bb78e8537f0c07946b014a65d32c27b791c8796ff0eefc94a95e06bb5621" exitCode=60
Oct 31 04:15:02.065353 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.065298    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l" event={"ID":"b13e4466-9375-4740-9ba7-12d286d0142a","Type":"ContainerDied","Data":"f472bb78e8537f0c07946b014a65d32c27b791c8796ff0eefc94a95e06bb5621"}
Oct 31 04:15:02.065617 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.065595    5415 scope.go:122] "RemoveContainer" containerID="f472bb78e8537f0c07946b014a65d32c27b791c8796ff0eefc94a95e06bb5621"
Oct 31 04:15:02.067308 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.067281    5415 generic.go:356] "Generic (PLEG): container finished" podID="6fa92af3-a192-4c80-bdb0-097b66fe9f14" containerID="38eebc86ced0847195100cda35337a6dd2e842376598b99542dcdccd303c27c8" exitCode=60
Oct 31 04:15:02.067415 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.067368    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c" event={"ID":"6fa92af3-a192-4c80-bdb0-097b66fe9f14","Type":"ContainerDied","Data":"38eebc86ced0847195100cda35337a6dd2e842376598b99542dcdccd303c27c8"}
Oct 31 04:15:02.067475 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.067429    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c" event={"ID":"6fa92af3-a192-4c80-bdb0-097b66fe9f14","Type":"ContainerStarted","Data":"754401d3e4f5328a215fc79e4ed53599249592e34c05279810fdfd8860f1abda"}
Oct 31 04:15:02.067629 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.067612    5415 scope.go:122] "RemoveContainer" containerID="38eebc86ced0847195100cda35337a6dd2e842376598b99542dcdccd303c27c8"
Oct 31 04:15:02.075305 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.075204    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq" event={"ID":"d8034d48-df5b-40ae-a7a0-c026aad438ed","Type":"ContainerStarted","Data":"0b39d61bc76109528e9b4ed9bdd37ac61220ac2dbdb9238eb95f58485b192de8"}
Oct 31 04:15:02.075305 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.075269    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq" event={"ID":"d8034d48-df5b-40ae-a7a0-c026aad438ed","Type":"ContainerStarted","Data":"c85ac2ccdcb6dbbbeb074ba266c81fb42f0bd14a42a6d65bd8e2b359478d8bbc"}
Oct 31 04:15:02.075513 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:02.075493    5415 scope.go:122] "RemoveContainer" containerID="0b39d61bc76109528e9b4ed9bdd37ac61220ac2dbdb9238eb95f58485b192de8"
Oct 31 04:15:03.083410 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.083377    5415 generic.go:356] "Generic (PLEG): container finished" podID="5e4f1595-2c8a-4408-a037-9de9b29669cb" containerID="a8117032f0403a71cb3fb202bd6a489bf3ceea0d9e7c678bf971da4e5527252f" exitCode=60
Oct 31 04:15:03.083935 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.083462    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b" event={"ID":"5e4f1595-2c8a-4408-a037-9de9b29669cb","Type":"ContainerDied","Data":"a8117032f0403a71cb3fb202bd6a489bf3ceea0d9e7c678bf971da4e5527252f"}
Oct 31 04:15:03.083935 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.083527    5415 scope.go:122] "RemoveContainer" containerID="8ca0f173608a7c72cebc33d60a877a501f81888be30dedc570bf0b87e8c3acaf"
Oct 31 04:15:03.083935 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.083819    5415 scope.go:122] "RemoveContainer" containerID="a8117032f0403a71cb3fb202bd6a489bf3ceea0d9e7c678bf971da4e5527252f"
Oct 31 04:15:03.084207 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:03.084183    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b_odh-ai-gateway-infra(5e4f1595-2c8a-4408-a037-9de9b29669cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b" podUID="5e4f1595-2c8a-4408-a037-9de9b29669cb"
Oct 31 04:15:03.085536 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.085517    5415 generic.go:356] "Generic (PLEG): container finished" podID="6a70215e-fc06-4236-91cb-bda11d72454b" containerID="ee04d61e5432bc390ed9ccf7604b1169d3c2fe5dbf73d4b953f46d97bd694ceb" exitCode=60
Oct 31 04:15:03.085646 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.085603    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr" event={"ID":"6a70215e-fc06-4236-91cb-bda11d72454b","Type":"ContainerDied","Data":"ee04d61e5432bc390ed9ccf7604b1169d3c2fe5dbf73d4b953f46d97bd694ceb"}
Oct 31 04:15:03.085710 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.085663    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr" event={"ID":"6a70215e-fc06-4236-91cb-bda11d72454b","Type":"ContainerStarted","Data":"da45b66c253fae03a6dab69aef6a03b2b6a9f35bf8e813e8267482d21b0fd18e"}
Oct 31 04:15:03.087597 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.087562    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l" event={"ID":"b13e4466-9375-4740-9ba7-12d286d0142a","Type":"ContainerStarted","Data":"d9c6103db3a15563800346d72fe91ec44c55a01be87f2b2ddae6f8d38e8552f1"}
Oct 31 04:15:03.089570 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.089549    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c" event={"ID":"6fa92af3-a192-4c80-bdb0-097b66fe9f14","Type":"ContainerStarted","Data":"84c608bf33125a8b4cce247bcc698c843dc1403eabc6e06b6a91690f6144f664"}
Oct 31 04:15:03.091444 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.091414    5415 generic.go:356] "Generic (PLEG): container finished" podID="d8034d48-df5b-40ae-a7a0-c026aad438ed" containerID="0b39d61bc76109528e9b4ed9bdd37ac61220ac2dbdb9238eb95f58485b192de8" exitCode=60
Oct 31 04:15:03.091544 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.091458    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq" event={"ID":"d8034d48-df5b-40ae-a7a0-c026aad438ed","Type":"ContainerDied","Data":"0b39d61bc76109528e9b4ed9bdd37ac61220ac2dbdb9238eb95f58485b192de8"}
Oct 31 04:15:03.091544 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.091491    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq" event={"ID":"d8034d48-df5b-40ae-a7a0-c026aad438ed","Type":"ContainerStarted","Data":"c18690381bb9749cafe9f6b1b8f185651b94d1abfee83225f334e5b7b834fed3"}
Oct 31 04:15:03.298273 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.298175    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr" podStartSLOduration=3.298158045 podStartE2EDuration="3.298158045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:03.29758762 +0000 UTC m=+4044.858067372" watchObservedRunningTime="2026-10-31 04:15:03.298158045 +0000 UTC m=+4044.858637776"
Oct 31 04:15:03.404325 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.404269    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c" podStartSLOduration=3.404255293 podStartE2EDuration="3.404255293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:03.404075445 +0000 UTC m=+4044.964555195" watchObservedRunningTime="2026-10-31 04:15:03.404255293 +0000 UTC m=+4044.964735023"
Oct 31 04:15:03.511990 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.511928    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l" podStartSLOduration=3.511912682 podStartE2EDuration="3.511912682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:03.510568178 +0000 UTC m=+4045.071047909" watchObservedRunningTime="2026-10-31 04:15:03.511912682 +0000 UTC m=+4045.072392412"
Oct 31 04:15:03.617782 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:03.617727    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq" podStartSLOduration=3.617711847 podStartE2EDuration="3.617711847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:03.617043091 +0000 UTC m=+4045.177522823" watchObservedRunningTime="2026-10-31 04:15:03.617711847 +0000 UTC m=+4045.178191578"
Oct 31 04:15:04.099733 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.099700    5415 generic.go:356] "Generic (PLEG): container finished" podID="6fa92af3-a192-4c80-bdb0-097b66fe9f14" containerID="84c608bf33125a8b4cce247bcc698c843dc1403eabc6e06b6a91690f6144f664" exitCode=60
Oct 31 04:15:04.100256 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.099793    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c" event={"ID":"6fa92af3-a192-4c80-bdb0-097b66fe9f14","Type":"ContainerDied","Data":"84c608bf33125a8b4cce247bcc698c843dc1403eabc6e06b6a91690f6144f664"}
Oct 31 04:15:04.100256 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.099862    5415 scope.go:122] "RemoveContainer" containerID="38eebc86ced0847195100cda35337a6dd2e842376598b99542dcdccd303c27c8"
Oct 31 04:15:04.100256 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.100191    5415 scope.go:122] "RemoveContainer" containerID="84c608bf33125a8b4cce247bcc698c843dc1403eabc6e06b6a91690f6144f664"
Oct 31 04:15:04.100522 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:04.100498    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c_odh-ai-gateway-infra(6fa92af3-a192-4c80-bdb0-097b66fe9f14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c" podUID="6fa92af3-a192-4c80-bdb0-097b66fe9f14"
Oct 31 04:15:04.102164 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.102145    5415 generic.go:356] "Generic (PLEG): container finished" podID="d8034d48-df5b-40ae-a7a0-c026aad438ed" containerID="c18690381bb9749cafe9f6b1b8f185651b94d1abfee83225f334e5b7b834fed3" exitCode=60
Oct 31 04:15:04.102260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.102232    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq" event={"ID":"d8034d48-df5b-40ae-a7a0-c026aad438ed","Type":"ContainerDied","Data":"c18690381bb9749cafe9f6b1b8f185651b94d1abfee83225f334e5b7b834fed3"}
Oct 31 04:15:04.102481 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.102466    5415 scope.go:122] "RemoveContainer" containerID="c18690381bb9749cafe9f6b1b8f185651b94d1abfee83225f334e5b7b834fed3"
Oct 31 04:15:04.102813 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:04.102784    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq_odh-ai-gateway-infra(d8034d48-df5b-40ae-a7a0-c026aad438ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq" podUID="d8034d48-df5b-40ae-a7a0-c026aad438ed"
Oct 31 04:15:04.104258 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.104237    5415 scope.go:122] "RemoveContainer" containerID="a8117032f0403a71cb3fb202bd6a489bf3ceea0d9e7c678bf971da4e5527252f"
Oct 31 04:15:04.104544 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:04.104521    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b_odh-ai-gateway-infra(5e4f1595-2c8a-4408-a037-9de9b29669cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b" podUID="5e4f1595-2c8a-4408-a037-9de9b29669cb"
Oct 31 04:15:04.105927 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.105869    5415 generic.go:356] "Generic (PLEG): container finished" podID="6a70215e-fc06-4236-91cb-bda11d72454b" containerID="da45b66c253fae03a6dab69aef6a03b2b6a9f35bf8e813e8267482d21b0fd18e" exitCode=60
Oct 31 04:15:04.106031 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.105909    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr" event={"ID":"6a70215e-fc06-4236-91cb-bda11d72454b","Type":"ContainerDied","Data":"da45b66c253fae03a6dab69aef6a03b2b6a9f35bf8e813e8267482d21b0fd18e"}
Oct 31 04:15:04.106221 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.106206    5415 scope.go:122] "RemoveContainer" containerID="da45b66c253fae03a6dab69aef6a03b2b6a9f35bf8e813e8267482d21b0fd18e"
Oct 31 04:15:04.106541 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:04.106515    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr_odh-ai-gateway-infra(6a70215e-fc06-4236-91cb-bda11d72454b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr" podUID="6a70215e-fc06-4236-91cb-bda11d72454b"
Oct 31 04:15:04.107930 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.107910    5415 generic.go:356] "Generic (PLEG): container finished" podID="b13e4466-9375-4740-9ba7-12d286d0142a" containerID="d9c6103db3a15563800346d72fe91ec44c55a01be87f2b2ddae6f8d38e8552f1" exitCode=60
Oct 31 04:15:04.108019 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.107965    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l" event={"ID":"b13e4466-9375-4740-9ba7-12d286d0142a","Type":"ContainerDied","Data":"d9c6103db3a15563800346d72fe91ec44c55a01be87f2b2ddae6f8d38e8552f1"}
Oct 31 04:15:04.108240 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.108226    5415 scope.go:122] "RemoveContainer" containerID="d9c6103db3a15563800346d72fe91ec44c55a01be87f2b2ddae6f8d38e8552f1"
Oct 31 04:15:04.108512 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:04.108491    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l_odh-ai-gateway-infra(b13e4466-9375-4740-9ba7-12d286d0142a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l" podUID="b13e4466-9375-4740-9ba7-12d286d0142a"
Oct 31 04:15:04.114071 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.114051    5415 scope.go:122] "RemoveContainer" containerID="0b39d61bc76109528e9b4ed9bdd37ac61220ac2dbdb9238eb95f58485b192de8"
Oct 31 04:15:04.127892 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.127857    5415 scope.go:122] "RemoveContainer" containerID="ee04d61e5432bc390ed9ccf7604b1169d3c2fe5dbf73d4b953f46d97bd694ceb"
Oct 31 04:15:04.144296 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:04.144254    5415 scope.go:122] "RemoveContainer" containerID="f472bb78e8537f0c07946b014a65d32c27b791c8796ff0eefc94a95e06bb5621"
Oct 31 04:15:05.115903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:05.115848    5415 scope.go:122] "RemoveContainer" containerID="d9c6103db3a15563800346d72fe91ec44c55a01be87f2b2ddae6f8d38e8552f1"
Oct 31 04:15:05.116433 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:05.116181    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l_odh-ai-gateway-infra(b13e4466-9375-4740-9ba7-12d286d0142a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l" podUID="b13e4466-9375-4740-9ba7-12d286d0142a"
Oct 31 04:15:05.117617 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:05.117594    5415 scope.go:122] "RemoveContainer" containerID="84c608bf33125a8b4cce247bcc698c843dc1403eabc6e06b6a91690f6144f664"
Oct 31 04:15:05.117917 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:05.117889    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c_odh-ai-gateway-infra(6fa92af3-a192-4c80-bdb0-097b66fe9f14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c" podUID="6fa92af3-a192-4c80-bdb0-097b66fe9f14"
Oct 31 04:15:05.119132 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:05.119114    5415 scope.go:122] "RemoveContainer" containerID="c18690381bb9749cafe9f6b1b8f185651b94d1abfee83225f334e5b7b834fed3"
Oct 31 04:15:05.119392 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:05.119365    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq_odh-ai-gateway-infra(d8034d48-df5b-40ae-a7a0-c026aad438ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq" podUID="d8034d48-df5b-40ae-a7a0-c026aad438ed"
Oct 31 04:15:05.120680 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:05.120661    5415 scope.go:122] "RemoveContainer" containerID="da45b66c253fae03a6dab69aef6a03b2b6a9f35bf8e813e8267482d21b0fd18e"
Oct 31 04:15:05.120957 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:05.120936    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr_odh-ai-gateway-infra(6a70215e-fc06-4236-91cb-bda11d72454b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr" podUID="6a70215e-fc06-4236-91cb-bda11d72454b"
Oct 31 04:15:15.598994 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:15.598915    5415 scope.go:122] "RemoveContainer" containerID="d9c6103db3a15563800346d72fe91ec44c55a01be87f2b2ddae6f8d38e8552f1"
Oct 31 04:15:15.599511 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:15.598992    5415 scope.go:122] "RemoveContainer" containerID="c18690381bb9749cafe9f6b1b8f185651b94d1abfee83225f334e5b7b834fed3"
Oct 31 04:15:16.177376 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:16.177340    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:15:16.192859 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:16.192817    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l" event={"ID":"b13e4466-9375-4740-9ba7-12d286d0142a","Type":"ContainerStarted","Data":"46363e5bb66bf0ef89f030e84f21b02331523b14340ab7deb8e8e51f349e736e"}
Oct 31 04:15:16.195283 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:16.195257    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq" event={"ID":"d8034d48-df5b-40ae-a7a0-c026aad438ed","Type":"ContainerStarted","Data":"b407ffaeec1a922b8e5fc19ce97995613519115498486c83f5cd7d224cf0f463"}
Oct 31 04:15:16.599001 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:16.598970    5415 scope.go:122] "RemoveContainer" containerID="da45b66c253fae03a6dab69aef6a03b2b6a9f35bf8e813e8267482d21b0fd18e"
Oct 31 04:15:16.599492 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:16.599114    5415 scope.go:122] "RemoveContainer" containerID="a8117032f0403a71cb3fb202bd6a489bf3ceea0d9e7c678bf971da4e5527252f"
Oct 31 04:15:17.203063 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.203026    5415 generic.go:356] "Generic (PLEG): container finished" podID="d8034d48-df5b-40ae-a7a0-c026aad438ed" containerID="b407ffaeec1a922b8e5fc19ce97995613519115498486c83f5cd7d224cf0f463" exitCode=60
Oct 31 04:15:17.203063 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.203052    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq" event={"ID":"d8034d48-df5b-40ae-a7a0-c026aad438ed","Type":"ContainerDied","Data":"b407ffaeec1a922b8e5fc19ce97995613519115498486c83f5cd7d224cf0f463"}
Oct 31 04:15:17.203283 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.203119    5415 scope.go:122] "RemoveContainer" containerID="c18690381bb9749cafe9f6b1b8f185651b94d1abfee83225f334e5b7b834fed3"
Oct 31 04:15:17.203524 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.203498    5415 scope.go:122] "RemoveContainer" containerID="b407ffaeec1a922b8e5fc19ce97995613519115498486c83f5cd7d224cf0f463"
Oct 31 04:15:17.203903 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:17.203859    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq_odh-ai-gateway-infra(d8034d48-df5b-40ae-a7a0-c026aad438ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq" podUID="d8034d48-df5b-40ae-a7a0-c026aad438ed"
Oct 31 04:15:17.205460 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.205433    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b" event={"ID":"5e4f1595-2c8a-4408-a037-9de9b29669cb","Type":"ContainerStarted","Data":"4c860fe73c0706cea47d3dcc4b85d4588ed87cfad6bf2473c311a72303712f37"}
Oct 31 04:15:17.207399 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.207374    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr" event={"ID":"6a70215e-fc06-4236-91cb-bda11d72454b","Type":"ContainerStarted","Data":"e06031c07b00569959a95ed60784277ebde6ccc07a966ad4ba3d8a3045c2a855"}
Oct 31 04:15:17.209500 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.209474    5415 generic.go:356] "Generic (PLEG): container finished" podID="b13e4466-9375-4740-9ba7-12d286d0142a" containerID="46363e5bb66bf0ef89f030e84f21b02331523b14340ab7deb8e8e51f349e736e" exitCode=60
Oct 31 04:15:17.209619 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.209523    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l" event={"ID":"b13e4466-9375-4740-9ba7-12d286d0142a","Type":"ContainerDied","Data":"46363e5bb66bf0ef89f030e84f21b02331523b14340ab7deb8e8e51f349e736e"}
Oct 31 04:15:17.209860 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.209844    5415 scope.go:122] "RemoveContainer" containerID="46363e5bb66bf0ef89f030e84f21b02331523b14340ab7deb8e8e51f349e736e"
Oct 31 04:15:17.210147 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:17.210125    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l_odh-ai-gateway-infra(b13e4466-9375-4740-9ba7-12d286d0142a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l" podUID="b13e4466-9375-4740-9ba7-12d286d0142a"
Oct 31 04:15:17.216465 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.216445    5415 scope.go:122] "RemoveContainer" containerID="d9c6103db3a15563800346d72fe91ec44c55a01be87f2b2ddae6f8d38e8552f1"
Oct 31 04:15:17.310217 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.310168    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"]
Oct 31 04:15:17.417934 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.417761    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"]
Oct 31 04:15:17.526079 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.526026    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b" podStartSLOduration=17.526013174 podStartE2EDuration="17.526013174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:17.525183945 +0000 UTC m=+4059.085663679" watchObservedRunningTime="2026-10-31 04:15:17.526013174 +0000 UTC m=+4059.086492905"
Oct 31 04:15:17.599510 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:17.599483    5415 scope.go:122] "RemoveContainer" containerID="84c608bf33125a8b4cce247bcc698c843dc1403eabc6e06b6a91690f6144f664"
Oct 31 04:15:18.219161 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.219128    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c" event={"ID":"6fa92af3-a192-4c80-bdb0-097b66fe9f14","Type":"ContainerStarted","Data":"9fe568df02ba1a05fd1d1db5c2b078d57307a420abfede5ffe55da3f99e471cb"}
Oct 31 04:15:18.223062 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.223036    5415 generic.go:356] "Generic (PLEG): container finished" podID="5e4f1595-2c8a-4408-a037-9de9b29669cb" containerID="4c860fe73c0706cea47d3dcc4b85d4588ed87cfad6bf2473c311a72303712f37" exitCode=60
Oct 31 04:15:18.223193 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.223113    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b" event={"ID":"5e4f1595-2c8a-4408-a037-9de9b29669cb","Type":"ContainerDied","Data":"4c860fe73c0706cea47d3dcc4b85d4588ed87cfad6bf2473c311a72303712f37"}
Oct 31 04:15:18.223193 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.223180    5415 scope.go:122] "RemoveContainer" containerID="a8117032f0403a71cb3fb202bd6a489bf3ceea0d9e7c678bf971da4e5527252f"
Oct 31 04:15:18.223568 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.223548    5415 scope.go:122] "RemoveContainer" containerID="4c860fe73c0706cea47d3dcc4b85d4588ed87cfad6bf2473c311a72303712f37"
Oct 31 04:15:18.223958 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:18.223916    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b_odh-ai-gateway-infra(5e4f1595-2c8a-4408-a037-9de9b29669cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b" podUID="5e4f1595-2c8a-4408-a037-9de9b29669cb"
Oct 31 04:15:18.225549 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.225525    5415 generic.go:356] "Generic (PLEG): container finished" podID="6a70215e-fc06-4236-91cb-bda11d72454b" containerID="e06031c07b00569959a95ed60784277ebde6ccc07a966ad4ba3d8a3045c2a855" exitCode=60
Oct 31 04:15:18.225643 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.225563    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr" event={"ID":"6a70215e-fc06-4236-91cb-bda11d72454b","Type":"ContainerDied","Data":"e06031c07b00569959a95ed60784277ebde6ccc07a966ad4ba3d8a3045c2a855"}
Oct 31 04:15:18.226010 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.225991    5415 scope.go:122] "RemoveContainer" containerID="e06031c07b00569959a95ed60784277ebde6ccc07a966ad4ba3d8a3045c2a855"
Oct 31 04:15:18.226357 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:18.226334    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr_odh-ai-gateway-infra(6a70215e-fc06-4236-91cb-bda11d72454b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr" podUID="6a70215e-fc06-4236-91cb-bda11d72454b"
Oct 31 04:15:18.239465 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.238267    5415 scope.go:122] "RemoveContainer" containerID="da45b66c253fae03a6dab69aef6a03b2b6a9f35bf8e813e8267482d21b0fd18e"
Oct 31 04:15:18.390232 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.390198    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"
Oct 31 04:15:18.430381 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.430353    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"
Oct 31 04:15:18.476749 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.476653    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8034d48-df5b-40ae-a7a0-c026aad438ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8034d48-df5b-40ae-a7a0-c026aad438ed-maas-api-service-ca\") pod \"d8034d48-df5b-40ae-a7a0-c026aad438ed\" (UID: \"d8034d48-df5b-40ae-a7a0-c026aad438ed\") "
Oct 31 04:15:18.476749 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.476703    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b13e4466-9375-4740-9ba7-12d286d0142a-kube-api-access-4pcls\" (UniqueName: \"kubernetes.io/projected/b13e4466-9375-4740-9ba7-12d286d0142a-kube-api-access-4pcls\") pod \"b13e4466-9375-4740-9ba7-12d286d0142a\" (UID: \"b13e4466-9375-4740-9ba7-12d286d0142a\") "
Oct 31 04:15:18.476749 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.476725    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8034d48-df5b-40ae-a7a0-c026aad438ed-kube-api-access-jc5z2\" (UniqueName: \"kubernetes.io/projected/d8034d48-df5b-40ae-a7a0-c026aad438ed-kube-api-access-jc5z2\") pod \"d8034d48-df5b-40ae-a7a0-c026aad438ed\" (UID: \"d8034d48-df5b-40ae-a7a0-c026aad438ed\") "
Oct 31 04:15:18.477088 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.476820    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b13e4466-9375-4740-9ba7-12d286d0142a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13e4466-9375-4740-9ba7-12d286d0142a-maas-api-service-ca\") pod \"b13e4466-9375-4740-9ba7-12d286d0142a\" (UID: \"b13e4466-9375-4740-9ba7-12d286d0142a\") "
Oct 31 04:15:18.477220 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.477188    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8034d48-df5b-40ae-a7a0-c026aad438ed-maas-api-service-ca" pod "d8034d48-df5b-40ae-a7a0-c026aad438ed" (UID: "d8034d48-df5b-40ae-a7a0-c026aad438ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:18.477315 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.477284    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b13e4466-9375-4740-9ba7-12d286d0142a-maas-api-service-ca" pod "b13e4466-9375-4740-9ba7-12d286d0142a" (UID: "b13e4466-9375-4740-9ba7-12d286d0142a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:18.477423 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.477395    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8034d48-df5b-40ae-a7a0-c026aad438ed-maas-api-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:18.477482 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.477429    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13e4466-9375-4740-9ba7-12d286d0142a-maas-api-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:18.478978 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.478957    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8034d48-df5b-40ae-a7a0-c026aad438ed-kube-api-access-jc5z2" pod "d8034d48-df5b-40ae-a7a0-c026aad438ed" (UID: "d8034d48-df5b-40ae-a7a0-c026aad438ed"). InnerVolumeSpecName "kube-api-access-jc5z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:18.479048 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.478998    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13e4466-9375-4740-9ba7-12d286d0142a-kube-api-access-4pcls" pod "b13e4466-9375-4740-9ba7-12d286d0142a" (UID: "b13e4466-9375-4740-9ba7-12d286d0142a"). InnerVolumeSpecName "kube-api-access-4pcls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:18.534815 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.534783    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"]
Oct 31 04:15:18.579138 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.579101    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pcls\" (UniqueName: \"kubernetes.io/projected/b13e4466-9375-4740-9ba7-12d286d0142a-kube-api-access-4pcls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:18.579138 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.579130    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc5z2\" (UniqueName: \"kubernetes.io/projected/d8034d48-df5b-40ae-a7a0-c026aad438ed-kube-api-access-jc5z2\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:18.641915 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:18.641865    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"]
Oct 31 04:15:19.236357 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.236332    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"
Oct 31 04:15:19.236498 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.236364    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l" event={"ID":"b13e4466-9375-4740-9ba7-12d286d0142a","Type":"ContainerDied","Data":"a7cd3996280308cfb8eeac1ef75d44b7e5c9020da52e4f3fcf09a4930853a23d"}
Oct 31 04:15:19.236498 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.236438    5415 scope.go:122] "RemoveContainer" containerID="46363e5bb66bf0ef89f030e84f21b02331523b14340ab7deb8e8e51f349e736e"
Oct 31 04:15:19.240394 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.240365    5415 generic.go:356] "Generic (PLEG): container finished" podID="6fa92af3-a192-4c80-bdb0-097b66fe9f14" containerID="9fe568df02ba1a05fd1d1db5c2b078d57307a420abfede5ffe55da3f99e471cb" exitCode=60
Oct 31 04:15:19.240525 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.240433    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c" event={"ID":"6fa92af3-a192-4c80-bdb0-097b66fe9f14","Type":"ContainerDied","Data":"9fe568df02ba1a05fd1d1db5c2b078d57307a420abfede5ffe55da3f99e471cb"}
Oct 31 04:15:19.240925 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.240903    5415 scope.go:122] "RemoveContainer" containerID="9fe568df02ba1a05fd1d1db5c2b078d57307a420abfede5ffe55da3f99e471cb"
Oct 31 04:15:19.241269 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:19.241244    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c_odh-ai-gateway-infra(6fa92af3-a192-4c80-bdb0-097b66fe9f14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c" podUID="6fa92af3-a192-4c80-bdb0-097b66fe9f14"
Oct 31 04:15:19.242213 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.242187    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"
Oct 31 04:15:19.242333 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.242230    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq" event={"ID":"d8034d48-df5b-40ae-a7a0-c026aad438ed","Type":"ContainerDied","Data":"c85ac2ccdcb6dbbbeb074ba266c81fb42f0bd14a42a6d65bd8e2b359478d8bbc"}
Oct 31 04:15:19.249589 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.249049    5415 scope.go:122] "RemoveContainer" containerID="84c608bf33125a8b4cce247bcc698c843dc1403eabc6e06b6a91690f6144f664"
Oct 31 04:15:19.266629 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.266272    5415 scope.go:122] "RemoveContainer" containerID="b407ffaeec1a922b8e5fc19ce97995613519115498486c83f5cd7d224cf0f463"
Oct 31 04:15:19.399733 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.399703    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"
Oct 31 04:15:19.445746 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.445718    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"
Oct 31 04:15:19.448287 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.448259    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"]
Oct 31 04:15:19.452115 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.452092    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d247af-29890335-nll5l"]
Oct 31 04:15:19.489038 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.488945    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4f1595-2c8a-4408-a037-9de9b29669cb-kube-api-access-jt7fb\" (UniqueName: \"kubernetes.io/projected/5e4f1595-2c8a-4408-a037-9de9b29669cb-kube-api-access-jt7fb\") pod \"5e4f1595-2c8a-4408-a037-9de9b29669cb\" (UID: \"5e4f1595-2c8a-4408-a037-9de9b29669cb\") "
Oct 31 04:15:19.489203 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.489042    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a70215e-fc06-4236-91cb-bda11d72454b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a70215e-fc06-4236-91cb-bda11d72454b-maas-api-service-ca\") pod \"6a70215e-fc06-4236-91cb-bda11d72454b\" (UID: \"6a70215e-fc06-4236-91cb-bda11d72454b\") "
Oct 31 04:15:19.489203 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.489080    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e4f1595-2c8a-4408-a037-9de9b29669cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f1595-2c8a-4408-a037-9de9b29669cb-maas-api-service-ca\") pod \"5e4f1595-2c8a-4408-a037-9de9b29669cb\" (UID: \"5e4f1595-2c8a-4408-a037-9de9b29669cb\") "
Oct 31 04:15:19.489203 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.489190    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a70215e-fc06-4236-91cb-bda11d72454b-kube-api-access-hlmxs\" (UniqueName: \"kubernetes.io/projected/6a70215e-fc06-4236-91cb-bda11d72454b-kube-api-access-hlmxs\") pod \"6a70215e-fc06-4236-91cb-bda11d72454b\" (UID: \"6a70215e-fc06-4236-91cb-bda11d72454b\") "
Oct 31 04:15:19.489483 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.489453    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a70215e-fc06-4236-91cb-bda11d72454b-maas-api-service-ca" pod "6a70215e-fc06-4236-91cb-bda11d72454b" (UID: "6a70215e-fc06-4236-91cb-bda11d72454b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:19.489539 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.489474    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4f1595-2c8a-4408-a037-9de9b29669cb-maas-api-service-ca" pod "5e4f1595-2c8a-4408-a037-9de9b29669cb" (UID: "5e4f1595-2c8a-4408-a037-9de9b29669cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:19.489736 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.489719    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a70215e-fc06-4236-91cb-bda11d72454b-maas-api-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:19.489782 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.489739    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f1595-2c8a-4408-a037-9de9b29669cb-maas-api-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:19.491296 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.491274    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4f1595-2c8a-4408-a037-9de9b29669cb-kube-api-access-jt7fb" pod "5e4f1595-2c8a-4408-a037-9de9b29669cb" (UID: "5e4f1595-2c8a-4408-a037-9de9b29669cb"). InnerVolumeSpecName "kube-api-access-jt7fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:19.491386 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.491300    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a70215e-fc06-4236-91cb-bda11d72454b-kube-api-access-hlmxs" pod "6a70215e-fc06-4236-91cb-bda11d72454b" (UID: "6a70215e-fc06-4236-91cb-bda11d72454b"). InnerVolumeSpecName "kube-api-access-hlmxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:19.568106 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.568072    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"]
Oct 31 04:15:19.574028 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.573986    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890335-mj2cq"]
Oct 31 04:15:19.590688 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.590656    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt7fb\" (UniqueName: \"kubernetes.io/projected/5e4f1595-2c8a-4408-a037-9de9b29669cb-kube-api-access-jt7fb\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:19.590688 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.590687    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlmxs\" (UniqueName: \"kubernetes.io/projected/6a70215e-fc06-4236-91cb-bda11d72454b-kube-api-access-hlmxs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:19.772736 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:19.772636    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"]
Oct 31 04:15:20.250081 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.250043    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"
Oct 31 04:15:20.250252 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.250046    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b" event={"ID":"5e4f1595-2c8a-4408-a037-9de9b29669cb","Type":"ContainerDied","Data":"7cc4a6ff0ee94b2f6252295eaedb51da3e55b18f92788a8b6de33fd76cd46866"}
Oct 31 04:15:20.250252 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.250197    5415 scope.go:122] "RemoveContainer" containerID="4c860fe73c0706cea47d3dcc4b85d4588ed87cfad6bf2473c311a72303712f37"
Oct 31 04:15:20.251630 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.251603    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr" event={"ID":"6a70215e-fc06-4236-91cb-bda11d72454b","Type":"ContainerDied","Data":"b29d0f7901395a2ba4dbead9699b62e9f98a6c5c9da194b789eb84d72302bd19"}
Oct 31 04:15:20.251726 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.251690    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"
Oct 31 04:15:20.263136 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.263114    5415 scope.go:122] "RemoveContainer" containerID="e06031c07b00569959a95ed60784277ebde6ccc07a966ad4ba3d8a3045c2a855"
Oct 31 04:15:20.405834 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.405800    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"]
Oct 31 04:15:20.409150 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.409121    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890335-stj2b"]
Oct 31 04:15:20.412562 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.412536    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"
Oct 31 04:15:20.502209 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.502110    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa92af3-a192-4c80-bdb0-097b66fe9f14-kube-api-access-nbwf7\" (UniqueName: \"kubernetes.io/projected/6fa92af3-a192-4c80-bdb0-097b66fe9f14-kube-api-access-nbwf7\") pod \"6fa92af3-a192-4c80-bdb0-097b66fe9f14\" (UID: \"6fa92af3-a192-4c80-bdb0-097b66fe9f14\") "
Oct 31 04:15:20.502367 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.502271    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa92af3-a192-4c80-bdb0-097b66fe9f14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa92af3-a192-4c80-bdb0-097b66fe9f14-maas-api-service-ca\") pod \"6fa92af3-a192-4c80-bdb0-097b66fe9f14\" (UID: \"6fa92af3-a192-4c80-bdb0-097b66fe9f14\") "
Oct 31 04:15:20.502652 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.502621    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa92af3-a192-4c80-bdb0-097b66fe9f14-maas-api-service-ca" pod "6fa92af3-a192-4c80-bdb0-097b66fe9f14" (UID: "6fa92af3-a192-4c80-bdb0-097b66fe9f14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:20.504465 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.504428    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa92af3-a192-4c80-bdb0-097b66fe9f14-kube-api-access-nbwf7" pod "6fa92af3-a192-4c80-bdb0-097b66fe9f14" (UID: "6fa92af3-a192-4c80-bdb0-097b66fe9f14"). InnerVolumeSpecName "kube-api-access-nbwf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:20.512676 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.512643    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"]
Oct 31 04:15:20.517681 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.517646    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890335-llvrr"]
Oct 31 04:15:20.605305 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.604972    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbwf7\" (UniqueName: \"kubernetes.io/projected/6fa92af3-a192-4c80-bdb0-097b66fe9f14-kube-api-access-nbwf7\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:20.605305 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.605008    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa92af3-a192-4c80-bdb0-097b66fe9f14-maas-api-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:20.611790 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.611759    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4f1595-2c8a-4408-a037-9de9b29669cb" path="/var/lib/kubelet/pods/5e4f1595-2c8a-4408-a037-9de9b29669cb/volumes"
Oct 31 04:15:20.612259 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.612241    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a70215e-fc06-4236-91cb-bda11d72454b" path="/var/lib/kubelet/pods/6a70215e-fc06-4236-91cb-bda11d72454b/volumes"
Oct 31 04:15:20.612612 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.612600    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b13e4466-9375-4740-9ba7-12d286d0142a" path="/var/lib/kubelet/pods/b13e4466-9375-4740-9ba7-12d286d0142a/volumes"
Oct 31 04:15:20.613280 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:20.613264    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8034d48-df5b-40ae-a7a0-c026aad438ed" path="/var/lib/kubelet/pods/d8034d48-df5b-40ae-a7a0-c026aad438ed/volumes"
Oct 31 04:15:21.266099 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:21.266066    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"
Oct 31 04:15:21.266553 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:21.266093    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c" event={"ID":"6fa92af3-a192-4c80-bdb0-097b66fe9f14","Type":"ContainerDied","Data":"754401d3e4f5328a215fc79e4ed53599249592e34c05279810fdfd8860f1abda"}
Oct 31 04:15:21.266553 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:21.266170    5415 scope.go:122] "RemoveContainer" containerID="9fe568df02ba1a05fd1d1db5c2b078d57307a420abfede5ffe55da3f99e471cb"
Oct 31 04:15:21.420819 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:21.420782    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"]
Oct 31 04:15:21.466574 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:21.466533    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f0e007-29890335-lcm8c"]
Oct 31 04:15:22.607051 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:15:22.607016    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa92af3-a192-4c80-bdb0-097b66fe9f14" path="/var/lib/kubelet/pods/6fa92af3-a192-4c80-bdb0-097b66fe9f14/volumes"
Oct 31 04:15:46.177212 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:15:46.177168    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:16:16.139128 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:16.139077    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"]
Oct 31 04:16:16.139685 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:16.139363    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv" podUID="83393f8e-2253-42a6-9fc0-25e5cc488155" containerName="istio-proxy" containerID="cri-o://57b46e09d8bb75d7a75cdd452fdb683781c5a0416f9cc626ff93ddb8f16dd980" gracePeriod=30
Oct 31 04:16:16.168967 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:16.168930    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"]
Oct 31 04:16:16.169418 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:16.169375    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb" podUID="96326289-4481-4634-b123-e93c01e0c1dc" containerName="istio-proxy" containerID="cri-o://05cd0de54992dd32b08024ba0883d15822afc56a781dcf1e27a0cbb3641e8133" gracePeriod=30
Oct 31 04:16:16.176948 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:16:16.176911    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:16:16.186061 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:16.186032    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"]
Oct 31 04:16:16.186254 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:16.186233    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy" containerID="cri-o://e982eb46517652004c8173bd7b7a453c7cb0c2600abe2dfe39643ec480451d11" gracePeriod=30
Oct 31 04:16:21.482185 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.482157    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:16:21.513131 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.513099    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:16:21.516538 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.516509    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:16:21.556689 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.556647    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-envoy\") pod \"83393f8e-2253-42a6-9fc0-25e5cc488155\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") "
Oct 31 04:16:21.556851 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.556745    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-kube-api-access-ffpqd\" (UniqueName: \"kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-kube-api-access-ffpqd\") pod \"83393f8e-2253-42a6-9fc0-25e5cc488155\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") "
Oct 31 04:16:21.556851 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.556812    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-credential-socket\") pod \"83393f8e-2253-42a6-9fc0-25e5cc488155\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") "
Oct 31 04:16:21.556851 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.556841    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-podinfo\") pod \"83393f8e-2253-42a6-9fc0-25e5cc488155\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") "
Oct 31 04:16:21.557073 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.556867    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-data\") pod \"83393f8e-2253-42a6-9fc0-25e5cc488155\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") "
Oct 31 04:16:21.557073 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.557028    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-certs\") pod \"83393f8e-2253-42a6-9fc0-25e5cc488155\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") "
Oct 31 04:16:21.557073 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.557054    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83393f8e-2253-42a6-9fc0-25e5cc488155-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83393f8e-2253-42a6-9fc0-25e5cc488155-istiod-ca-cert\") pod \"83393f8e-2253-42a6-9fc0-25e5cc488155\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") "
Oct 31 04:16:21.557297 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.557099    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-token\" (UniqueName: \"kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-token\") pod \"83393f8e-2253-42a6-9fc0-25e5cc488155\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") "
Oct 31 04:16:21.557297 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.557134    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-socket\") pod \"83393f8e-2253-42a6-9fc0-25e5cc488155\" (UID: \"83393f8e-2253-42a6-9fc0-25e5cc488155\") "
Oct 31 04:16:21.557420 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.557368    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-data" pod "83393f8e-2253-42a6-9fc0-25e5cc488155" (UID: "83393f8e-2253-42a6-9fc0-25e5cc488155"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.557743 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.557497    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83393f8e-2253-42a6-9fc0-25e5cc488155-istiod-ca-cert" pod "83393f8e-2253-42a6-9fc0-25e5cc488155" (UID: "83393f8e-2253-42a6-9fc0-25e5cc488155"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:16:21.557743 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.557595    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-socket" pod "83393f8e-2253-42a6-9fc0-25e5cc488155" (UID: "83393f8e-2253-42a6-9fc0-25e5cc488155"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.557743 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.557700    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-certs" pod "83393f8e-2253-42a6-9fc0-25e5cc488155" (UID: "83393f8e-2253-42a6-9fc0-25e5cc488155"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.558005 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.557871    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.558005 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.557930    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83393f8e-2253-42a6-9fc0-25e5cc488155-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.558005 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.557944    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.558005 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.557956    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.558165 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.558060    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-credential-socket" pod "83393f8e-2253-42a6-9fc0-25e5cc488155" (UID: "83393f8e-2253-42a6-9fc0-25e5cc488155"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.559726 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.559700    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-token" pod "83393f8e-2253-42a6-9fc0-25e5cc488155" (UID: "83393f8e-2253-42a6-9fc0-25e5cc488155"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:21.559850 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.559734    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-kube-api-access-ffpqd" pod "83393f8e-2253-42a6-9fc0-25e5cc488155" (UID: "83393f8e-2253-42a6-9fc0-25e5cc488155"). InnerVolumeSpecName "kube-api-access-ffpqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:21.559850 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.559770    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-envoy" pod "83393f8e-2253-42a6-9fc0-25e5cc488155" (UID: "83393f8e-2253-42a6-9fc0-25e5cc488155"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.559850 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.559790    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-podinfo" pod "83393f8e-2253-42a6-9fc0-25e5cc488155" (UID: "83393f8e-2253-42a6-9fc0-25e5cc488155"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:16:21.659314 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659280    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-credential-socket\") pod \"b31e83da-c659-4596-ac00-c7e4369911bf\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") "
Oct 31 04:16:21.659509 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659364    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-data\") pod \"96326289-4481-4634-b123-e93c01e0c1dc\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") "
Oct 31 04:16:21.659509 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659405    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-istio-token\" (UniqueName: \"kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-istio-token\") pod \"b31e83da-c659-4596-ac00-c7e4369911bf\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") "
Oct 31 04:16:21.659509 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659432    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-socket\") pod \"96326289-4481-4634-b123-e93c01e0c1dc\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") "
Oct 31 04:16:21.659509 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659458    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-envoy\") pod \"b31e83da-c659-4596-ac00-c7e4369911bf\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") "
Oct 31 04:16:21.659754 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659513    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96326289-4481-4634-b123-e93c01e0c1dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96326289-4481-4634-b123-e93c01e0c1dc-istiod-ca-cert\") pod \"96326289-4481-4634-b123-e93c01e0c1dc\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") "
Oct 31 04:16:21.659754 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659557    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-kube-api-access-dpd7m\" (UniqueName: \"kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-kube-api-access-dpd7m\") pod \"b31e83da-c659-4596-ac00-c7e4369911bf\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") "
Oct 31 04:16:21.659754 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659600    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-certs\") pod \"b31e83da-c659-4596-ac00-c7e4369911bf\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") "
Oct 31 04:16:21.659754 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659637    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-credential-socket" pod "b31e83da-c659-4596-ac00-c7e4369911bf" (UID: "b31e83da-c659-4596-ac00-c7e4369911bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.659754 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659673    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-data" pod "96326289-4481-4634-b123-e93c01e0c1dc" (UID: "96326289-4481-4634-b123-e93c01e0c1dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.659754 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659690    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-data\") pod \"b31e83da-c659-4596-ac00-c7e4369911bf\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") "
Oct 31 04:16:21.659754 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659741    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-kube-api-access-gm85x\" (UniqueName: \"kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-kube-api-access-gm85x\") pod \"96326289-4481-4634-b123-e93c01e0c1dc\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") "
Oct 31 04:16:21.660139 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659777    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96326289-4481-4634-b123-e93c01e0c1dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96326289-4481-4634-b123-e93c01e0c1dc-istio-podinfo\") pod \"96326289-4481-4634-b123-e93c01e0c1dc\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") "
Oct 31 04:16:21.660139 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659810    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-certs\") pod \"96326289-4481-4634-b123-e93c01e0c1dc\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") "
Oct 31 04:16:21.660139 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659841    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-istio-token\" (UniqueName: \"kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-istio-token\") pod \"96326289-4481-4634-b123-e93c01e0c1dc\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") "
Oct 31 04:16:21.660139 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659902    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b31e83da-c659-4596-ac00-c7e4369911bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31e83da-c659-4596-ac00-c7e4369911bf-istio-podinfo\") pod \"b31e83da-c659-4596-ac00-c7e4369911bf\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") "
Oct 31 04:16:21.660139 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659936    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31e83da-c659-4596-ac00-c7e4369911bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31e83da-c659-4596-ac00-c7e4369911bf-istiod-ca-cert\") pod \"b31e83da-c659-4596-ac00-c7e4369911bf\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") "
Oct 31 04:16:21.660139 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659944    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-socket" pod "96326289-4481-4634-b123-e93c01e0c1dc" (UID: "96326289-4481-4634-b123-e93c01e0c1dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.660139 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.659965    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-envoy\") pod \"96326289-4481-4634-b123-e93c01e0c1dc\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") "
Oct 31 04:16:21.660139 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660023    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-socket\") pod \"b31e83da-c659-4596-ac00-c7e4369911bf\" (UID: \"b31e83da-c659-4596-ac00-c7e4369911bf\") "
Oct 31 04:16:21.660139 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660051    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-credential-socket\") pod \"96326289-4481-4634-b123-e93c01e0c1dc\" (UID: \"96326289-4481-4634-b123-e93c01e0c1dc\") "
Oct 31 04:16:21.660139 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660091    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-data" pod "b31e83da-c659-4596-ac00-c7e4369911bf" (UID: "b31e83da-c659-4596-ac00-c7e4369911bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.660679 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660180    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-certs" pod "96326289-4481-4634-b123-e93c01e0c1dc" (UID: "96326289-4481-4634-b123-e93c01e0c1dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.660679 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660271    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-certs" pod "b31e83da-c659-4596-ac00-c7e4369911bf" (UID: "b31e83da-c659-4596-ac00-c7e4369911bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.660679 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660542    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31e83da-c659-4596-ac00-c7e4369911bf-istiod-ca-cert" pod "b31e83da-c659-4596-ac00-c7e4369911bf" (UID: "b31e83da-c659-4596-ac00-c7e4369911bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:16:21.660679 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660669    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.660924 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660690    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.660924 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660705    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.660924 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660720    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.660924 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660734    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.660924 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660748    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffpqd\" (UniqueName: \"kubernetes.io/projected/83393f8e-2253-42a6-9fc0-25e5cc488155-kube-api-access-ffpqd\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.660924 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660762    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.660924 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660776    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31e83da-c659-4596-ac00-c7e4369911bf-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.660924 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660791    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83393f8e-2253-42a6-9fc0-25e5cc488155-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.660924 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660805    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83393f8e-2253-42a6-9fc0-25e5cc488155-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.660924 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660820    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.660924 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.660836    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.662631 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.662603    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-kube-api-access-dpd7m" pod "b31e83da-c659-4596-ac00-c7e4369911bf" (UID: "b31e83da-c659-4596-ac00-c7e4369911bf"). InnerVolumeSpecName "kube-api-access-dpd7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:21.663018 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.662996    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96326289-4481-4634-b123-e93c01e0c1dc-istiod-ca-cert" pod "96326289-4481-4634-b123-e93c01e0c1dc" (UID: "96326289-4481-4634-b123-e93c01e0c1dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:16:21.663152 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.663132    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-kube-api-access-gm85x" pod "96326289-4481-4634-b123-e93c01e0c1dc" (UID: "96326289-4481-4634-b123-e93c01e0c1dc"). InnerVolumeSpecName "kube-api-access-gm85x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:21.663223 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.663159    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-istio-token" pod "b31e83da-c659-4596-ac00-c7e4369911bf" (UID: "b31e83da-c659-4596-ac00-c7e4369911bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:21.663223 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.663167    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-credential-socket" pod "96326289-4481-4634-b123-e93c01e0c1dc" (UID: "96326289-4481-4634-b123-e93c01e0c1dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.663339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.663237    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-socket" pod "b31e83da-c659-4596-ac00-c7e4369911bf" (UID: "b31e83da-c659-4596-ac00-c7e4369911bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.663791 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.663748    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-envoy" pod "b31e83da-c659-4596-ac00-c7e4369911bf" (UID: "b31e83da-c659-4596-ac00-c7e4369911bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.665098 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.665072    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-envoy" pod "96326289-4481-4634-b123-e93c01e0c1dc" (UID: "96326289-4481-4634-b123-e93c01e0c1dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.665196 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.665072    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-istio-token" pod "96326289-4481-4634-b123-e93c01e0c1dc" (UID: "96326289-4481-4634-b123-e93c01e0c1dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:21.665374 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.665351    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96326289-4481-4634-b123-e93c01e0c1dc-istio-podinfo" pod "96326289-4481-4634-b123-e93c01e0c1dc" (UID: "96326289-4481-4634-b123-e93c01e0c1dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:16:21.665430 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.665369    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b31e83da-c659-4596-ac00-c7e4369911bf-istio-podinfo" pod "b31e83da-c659-4596-ac00-c7e4369911bf" (UID: "b31e83da-c659-4596-ac00-c7e4369911bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:16:21.675164 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.675136    5415 generic.go:356] "Generic (PLEG): container finished" podID="96326289-4481-4634-b123-e93c01e0c1dc" containerID="05cd0de54992dd32b08024ba0883d15822afc56a781dcf1e27a0cbb3641e8133" exitCode=0
Oct 31 04:16:21.675303 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.675189    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"
Oct 31 04:16:21.675303 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.675217    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb" event={"ID":"96326289-4481-4634-b123-e93c01e0c1dc","Type":"ContainerDied","Data":"05cd0de54992dd32b08024ba0883d15822afc56a781dcf1e27a0cbb3641e8133"}
Oct 31 04:16:21.675303 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.675274    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb" event={"ID":"96326289-4481-4634-b123-e93c01e0c1dc","Type":"ContainerDied","Data":"74e7af8bd7cda8fc93d087f51cdb2a1d2e805b24fca3ee97ada441586d43417c"}
Oct 31 04:16:21.675470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.675304    5415 scope.go:122] "RemoveContainer" containerID="05cd0de54992dd32b08024ba0883d15822afc56a781dcf1e27a0cbb3641e8133"
Oct 31 04:16:21.677116 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.677092    5415 generic.go:356] "Generic (PLEG): container finished" podID="83393f8e-2253-42a6-9fc0-25e5cc488155" containerID="57b46e09d8bb75d7a75cdd452fdb683781c5a0416f9cc626ff93ddb8f16dd980" exitCode=0
Oct 31 04:16:21.677238 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.677149    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"
Oct 31 04:16:21.677238 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.677174    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv" event={"ID":"83393f8e-2253-42a6-9fc0-25e5cc488155","Type":"ContainerDied","Data":"57b46e09d8bb75d7a75cdd452fdb683781c5a0416f9cc626ff93ddb8f16dd980"}
Oct 31 04:16:21.677349 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.677247    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv" event={"ID":"83393f8e-2253-42a6-9fc0-25e5cc488155","Type":"ContainerDied","Data":"789e8f90871834cace8aa950dcfc92e621e9125a588405888b9964a5b64172a2"}
Oct 31 04:16:21.679176 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.679149    5415 generic.go:356] "Generic (PLEG): container finished" podID="b31e83da-c659-4596-ac00-c7e4369911bf" containerID="e982eb46517652004c8173bd7b7a453c7cb0c2600abe2dfe39643ec480451d11" exitCode=0
Oct 31 04:16:21.679309 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.679210    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" event={"ID":"b31e83da-c659-4596-ac00-c7e4369911bf","Type":"ContainerDied","Data":"e982eb46517652004c8173bd7b7a453c7cb0c2600abe2dfe39643ec480451d11"}
Oct 31 04:16:21.679309 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.679251    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s" event={"ID":"b31e83da-c659-4596-ac00-c7e4369911bf","Type":"ContainerDied","Data":"b2c3f554a12f0c7817a4282595d4362d4958c7e0e79081973567264a5cd0af78"}
Oct 31 04:16:21.679437 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.679334    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"
Oct 31 04:16:21.689982 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.689868    5415 scope.go:122] "RemoveContainer" containerID="05cd0de54992dd32b08024ba0883d15822afc56a781dcf1e27a0cbb3641e8133"
Oct 31 04:16:21.690395 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:16:21.690289    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05cd0de54992dd32b08024ba0883d15822afc56a781dcf1e27a0cbb3641e8133\": container with ID starting with 05cd0de54992dd32b08024ba0883d15822afc56a781dcf1e27a0cbb3641e8133 not found: ID does not exist" containerID="05cd0de54992dd32b08024ba0883d15822afc56a781dcf1e27a0cbb3641e8133"
Oct 31 04:16:21.690395 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.690328    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05cd0de54992dd32b08024ba0883d15822afc56a781dcf1e27a0cbb3641e8133"} err="failed to get container status \"05cd0de54992dd32b08024ba0883d15822afc56a781dcf1e27a0cbb3641e8133\": rpc error: code = NotFound desc = could not find container \"05cd0de54992dd32b08024ba0883d15822afc56a781dcf1e27a0cbb3641e8133\": container with ID starting with 05cd0de54992dd32b08024ba0883d15822afc56a781dcf1e27a0cbb3641e8133 not found: ID does not exist"
Oct 31 04:16:21.690395 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.690355    5415 scope.go:122] "RemoveContainer" containerID="57b46e09d8bb75d7a75cdd452fdb683781c5a0416f9cc626ff93ddb8f16dd980"
Oct 31 04:16:21.700532 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.700506    5415 scope.go:122] "RemoveContainer" containerID="57b46e09d8bb75d7a75cdd452fdb683781c5a0416f9cc626ff93ddb8f16dd980"
Oct 31 04:16:21.700823 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:16:21.700796    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b46e09d8bb75d7a75cdd452fdb683781c5a0416f9cc626ff93ddb8f16dd980\": container with ID starting with 57b46e09d8bb75d7a75cdd452fdb683781c5a0416f9cc626ff93ddb8f16dd980 not found: ID does not exist" containerID="57b46e09d8bb75d7a75cdd452fdb683781c5a0416f9cc626ff93ddb8f16dd980"
Oct 31 04:16:21.700897 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.700836    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b46e09d8bb75d7a75cdd452fdb683781c5a0416f9cc626ff93ddb8f16dd980"} err="failed to get container status \"57b46e09d8bb75d7a75cdd452fdb683781c5a0416f9cc626ff93ddb8f16dd980\": rpc error: code = NotFound desc = could not find container \"57b46e09d8bb75d7a75cdd452fdb683781c5a0416f9cc626ff93ddb8f16dd980\": container with ID starting with 57b46e09d8bb75d7a75cdd452fdb683781c5a0416f9cc626ff93ddb8f16dd980 not found: ID does not exist"
Oct 31 04:16:21.700897 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.700854    5415 scope.go:122] "RemoveContainer" containerID="e982eb46517652004c8173bd7b7a453c7cb0c2600abe2dfe39643ec480451d11"
Oct 31 04:16:21.710515 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.710494    5415 scope.go:122] "RemoveContainer" containerID="e982eb46517652004c8173bd7b7a453c7cb0c2600abe2dfe39643ec480451d11"
Oct 31 04:16:21.710786 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:16:21.710764    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e982eb46517652004c8173bd7b7a453c7cb0c2600abe2dfe39643ec480451d11\": container with ID starting with e982eb46517652004c8173bd7b7a453c7cb0c2600abe2dfe39643ec480451d11 not found: ID does not exist" containerID="e982eb46517652004c8173bd7b7a453c7cb0c2600abe2dfe39643ec480451d11"
Oct 31 04:16:21.710827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.710796    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e982eb46517652004c8173bd7b7a453c7cb0c2600abe2dfe39643ec480451d11"} err="failed to get container status \"e982eb46517652004c8173bd7b7a453c7cb0c2600abe2dfe39643ec480451d11\": rpc error: code = NotFound desc = could not find container \"e982eb46517652004c8173bd7b7a453c7cb0c2600abe2dfe39643ec480451d11\": container with ID starting with e982eb46517652004c8173bd7b7a453c7cb0c2600abe2dfe39643ec480451d11 not found: ID does not exist"
Oct 31 04:16:21.762140 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.762100    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.762140 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.762130    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96326289-4481-4634-b123-e93c01e0c1dc-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.762389 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.762147    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpd7m\" (UniqueName: \"kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-kube-api-access-dpd7m\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.762389 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.762164    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm85x\" (UniqueName: \"kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-kube-api-access-gm85x\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.762389 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.762173    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96326289-4481-4634-b123-e93c01e0c1dc-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.762389 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.762183    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96326289-4481-4634-b123-e93c01e0c1dc-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.762389 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.762192    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31e83da-c659-4596-ac00-c7e4369911bf-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.762389 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.762202    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.762389 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.762211    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e83da-c659-4596-ac00-c7e4369911bf-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.762389 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.762221    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96326289-4481-4634-b123-e93c01e0c1dc-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.762389 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.762231    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31e83da-c659-4596-ac00-c7e4369911bf-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.969255 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.969171    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"]
Oct 31 04:16:21.972333 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:21.972304    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vh5lb"]
Oct 31 04:16:22.076261 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:22.076225    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"]
Oct 31 04:16:22.081989 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:22.081955    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ppc6s"]
Oct 31 04:16:22.186530 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:22.186492    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"]
Oct 31 04:16:22.190174 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:22.190145    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jdrnv"]
Oct 31 04:16:22.606705 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:22.606670    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83393f8e-2253-42a6-9fc0-25e5cc488155" path="/var/lib/kubelet/pods/83393f8e-2253-42a6-9fc0-25e5cc488155/volumes"
Oct 31 04:16:22.607762 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:22.607720    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96326289-4481-4634-b123-e93c01e0c1dc" path="/var/lib/kubelet/pods/96326289-4481-4634-b123-e93c01e0c1dc/volumes"
Oct 31 04:16:22.608559 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:16:22.608535    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" path="/var/lib/kubelet/pods/b31e83da-c659-4596-ac00-c7e4369911bf/volumes"
Oct 31 04:16:46.175803 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:16:46.175725    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:17:16.176853 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:17:16.176809    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:17:22.464466 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:22.464426    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"]
Oct 31 04:17:23.693345 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:23.693306    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"]
Oct 31 04:17:23.693915 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:23.693627    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr" podUID="0af401f3-6429-4711-bcaf-56c6bba19998" containerName="maas-api" containerID="cri-o://e8f36e6145a67e44cddbf2a3144b921adcbdd3a2189e6d0c1b73ae903575ae3f" gracePeriod=30
Oct 31 04:17:23.776993 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:23.776953    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr" podUID="0af401f3-6429-4711-bcaf-56c6bba19998" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.138:8443/health\": dial tcp 10.134.0.138:8443: connect: connection refused"
Oct 31 04:17:23.979160 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:23.979125    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:17:24.040278 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.040238    5415 generic.go:356] "Generic (PLEG): container finished" podID="0af401f3-6429-4711-bcaf-56c6bba19998" containerID="e8f36e6145a67e44cddbf2a3144b921adcbdd3a2189e6d0c1b73ae903575ae3f" exitCode=0
Oct 31 04:17:24.040278 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.040283    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"
Oct 31 04:17:24.040278 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.040289    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr" event={"ID":"0af401f3-6429-4711-bcaf-56c6bba19998","Type":"ContainerDied","Data":"e8f36e6145a67e44cddbf2a3144b921adcbdd3a2189e6d0c1b73ae903575ae3f"}
Oct 31 04:17:24.040617 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.040340    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr" event={"ID":"0af401f3-6429-4711-bcaf-56c6bba19998","Type":"ContainerDied","Data":"3fde2edf4f7fb3a13e5167b1ae007af85c7e896920a77d2cf1dee0c6a81af90e"}
Oct 31 04:17:24.040617 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.040373    5415 scope.go:122] "RemoveContainer" containerID="e8f36e6145a67e44cddbf2a3144b921adcbdd3a2189e6d0c1b73ae903575ae3f"
Oct 31 04:17:24.053190 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.053158    5415 scope.go:122] "RemoveContainer" containerID="e8f36e6145a67e44cddbf2a3144b921adcbdd3a2189e6d0c1b73ae903575ae3f"
Oct 31 04:17:24.053556 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:17:24.053528    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f36e6145a67e44cddbf2a3144b921adcbdd3a2189e6d0c1b73ae903575ae3f\": container with ID starting with e8f36e6145a67e44cddbf2a3144b921adcbdd3a2189e6d0c1b73ae903575ae3f not found: ID does not exist" containerID="e8f36e6145a67e44cddbf2a3144b921adcbdd3a2189e6d0c1b73ae903575ae3f"
Oct 31 04:17:24.053682 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.053571    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f36e6145a67e44cddbf2a3144b921adcbdd3a2189e6d0c1b73ae903575ae3f"} err="failed to get container status \"e8f36e6145a67e44cddbf2a3144b921adcbdd3a2189e6d0c1b73ae903575ae3f\": rpc error: code = NotFound desc = could not find container \"e8f36e6145a67e44cddbf2a3144b921adcbdd3a2189e6d0c1b73ae903575ae3f\": container with ID starting with e8f36e6145a67e44cddbf2a3144b921adcbdd3a2189e6d0c1b73ae903575ae3f not found: ID does not exist"
Oct 31 04:17:24.082806 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.082766    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af401f3-6429-4711-bcaf-56c6bba19998-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af401f3-6429-4711-bcaf-56c6bba19998-openshift-service-ca\") pod \"0af401f3-6429-4711-bcaf-56c6bba19998\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") "
Oct 31 04:17:24.083054 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.082832    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-maas-api-tls\") pod \"0af401f3-6429-4711-bcaf-56c6bba19998\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") "
Oct 31 04:17:24.083054 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.082919    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af401f3-6429-4711-bcaf-56c6bba19998-kube-api-access-5q2wx\" (UniqueName: \"kubernetes.io/projected/0af401f3-6429-4711-bcaf-56c6bba19998-kube-api-access-5q2wx\") pod \"0af401f3-6429-4711-bcaf-56c6bba19998\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") "
Oct 31 04:17:24.083148 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.083126    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-metrics-certs\") pod \"0af401f3-6429-4711-bcaf-56c6bba19998\" (UID: \"0af401f3-6429-4711-bcaf-56c6bba19998\") "
Oct 31 04:17:24.083278 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.083248    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af401f3-6429-4711-bcaf-56c6bba19998-openshift-service-ca" pod "0af401f3-6429-4711-bcaf-56c6bba19998" (UID: "0af401f3-6429-4711-bcaf-56c6bba19998"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:17:24.083590 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.083565    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af401f3-6429-4711-bcaf-56c6bba19998-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:24.085339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.085292    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-metrics-certs" pod "0af401f3-6429-4711-bcaf-56c6bba19998" (UID: "0af401f3-6429-4711-bcaf-56c6bba19998"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:17:24.085446 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.085399    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af401f3-6429-4711-bcaf-56c6bba19998-kube-api-access-5q2wx" pod "0af401f3-6429-4711-bcaf-56c6bba19998" (UID: "0af401f3-6429-4711-bcaf-56c6bba19998"). InnerVolumeSpecName "kube-api-access-5q2wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:17:24.085509 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.085487    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-maas-api-tls" pod "0af401f3-6429-4711-bcaf-56c6bba19998" (UID: "0af401f3-6429-4711-bcaf-56c6bba19998"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:17:24.184750 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.184710    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:24.184750 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.184746    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af401f3-6429-4711-bcaf-56c6bba19998-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:24.184750 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.184757    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q2wx\" (UniqueName: \"kubernetes.io/projected/0af401f3-6429-4711-bcaf-56c6bba19998-kube-api-access-5q2wx\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:24.455898 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.455839    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"]
Oct 31 04:17:24.460706 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.460674    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-243a29-745bf7fc74-pdmfr"]
Oct 31 04:17:24.606605 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:24.606525    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af401f3-6429-4711-bcaf-56c6bba19998" path="/var/lib/kubelet/pods/0af401f3-6429-4711-bcaf-56c6bba19998/volumes"
Oct 31 04:17:27.465428 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:27.465370    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw" podUID="76bfbd25-939a-48b1-ae8e-e3cbf1313530" containerName="payload-processing" containerID="cri-o://43a68416f66811dbd1dbff15c737f036b70b5c8f80efeb52d51666a92becd7aa" gracePeriod=25
Oct 31 04:17:27.736895 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:27.736859    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"
Oct 31 04:17:27.827131 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:27.827091    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bfbd25-939a-48b1-ae8e-e3cbf1313530-kube-api-access-lglg2\" (UniqueName: \"kubernetes.io/projected/76bfbd25-939a-48b1-ae8e-e3cbf1313530-kube-api-access-lglg2\") pod \"76bfbd25-939a-48b1-ae8e-e3cbf1313530\" (UID: \"76bfbd25-939a-48b1-ae8e-e3cbf1313530\") "
Oct 31 04:17:27.827348 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:27.827139    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bfbd25-939a-48b1-ae8e-e3cbf1313530-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bfbd25-939a-48b1-ae8e-e3cbf1313530-plugins-config-volume\") pod \"76bfbd25-939a-48b1-ae8e-e3cbf1313530\" (UID: \"76bfbd25-939a-48b1-ae8e-e3cbf1313530\") "
Oct 31 04:17:27.827556 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:27.827532    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bfbd25-939a-48b1-ae8e-e3cbf1313530-plugins-config-volume" pod "76bfbd25-939a-48b1-ae8e-e3cbf1313530" (UID: "76bfbd25-939a-48b1-ae8e-e3cbf1313530"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:17:27.829379 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:27.829358    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bfbd25-939a-48b1-ae8e-e3cbf1313530-kube-api-access-lglg2" pod "76bfbd25-939a-48b1-ae8e-e3cbf1313530" (UID: "76bfbd25-939a-48b1-ae8e-e3cbf1313530"). InnerVolumeSpecName "kube-api-access-lglg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:17:27.928558 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:27.928516    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lglg2\" (UniqueName: \"kubernetes.io/projected/76bfbd25-939a-48b1-ae8e-e3cbf1313530-kube-api-access-lglg2\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:27.928558 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:27.928547    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bfbd25-939a-48b1-ae8e-e3cbf1313530-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:28.065815 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:28.065727    5415 generic.go:356] "Generic (PLEG): container finished" podID="76bfbd25-939a-48b1-ae8e-e3cbf1313530" containerID="43a68416f66811dbd1dbff15c737f036b70b5c8f80efeb52d51666a92becd7aa" exitCode=0
Oct 31 04:17:28.065815 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:28.065774    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"
Oct 31 04:17:28.066026 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:28.065818    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw" event={"ID":"76bfbd25-939a-48b1-ae8e-e3cbf1313530","Type":"ContainerDied","Data":"43a68416f66811dbd1dbff15c737f036b70b5c8f80efeb52d51666a92becd7aa"}
Oct 31 04:17:28.066026 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:28.065914    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw" event={"ID":"76bfbd25-939a-48b1-ae8e-e3cbf1313530","Type":"ContainerDied","Data":"f2a2bcac0fd4b5766b77fe66cad0395a31273463c9fda8d0bec94f6b96ca067d"}
Oct 31 04:17:28.066026 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:28.065940    5415 scope.go:122] "RemoveContainer" containerID="43a68416f66811dbd1dbff15c737f036b70b5c8f80efeb52d51666a92becd7aa"
Oct 31 04:17:28.076049 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:28.076030    5415 scope.go:122] "RemoveContainer" containerID="43a68416f66811dbd1dbff15c737f036b70b5c8f80efeb52d51666a92becd7aa"
Oct 31 04:17:28.076377 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:17:28.076357    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a68416f66811dbd1dbff15c737f036b70b5c8f80efeb52d51666a92becd7aa\": container with ID starting with 43a68416f66811dbd1dbff15c737f036b70b5c8f80efeb52d51666a92becd7aa not found: ID does not exist" containerID="43a68416f66811dbd1dbff15c737f036b70b5c8f80efeb52d51666a92becd7aa"
Oct 31 04:17:28.076433 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:28.076389    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a68416f66811dbd1dbff15c737f036b70b5c8f80efeb52d51666a92becd7aa"} err="failed to get container status \"43a68416f66811dbd1dbff15c737f036b70b5c8f80efeb52d51666a92becd7aa\": rpc error: code = NotFound desc = could not find container \"43a68416f66811dbd1dbff15c737f036b70b5c8f80efeb52d51666a92becd7aa\": container with ID starting with 43a68416f66811dbd1dbff15c737f036b70b5c8f80efeb52d51666a92becd7aa not found: ID does not exist"
Oct 31 04:17:28.220515 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:28.220481    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"]
Oct 31 04:17:28.223806 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:28.223778    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-243a29-dd8dddb6f-7kfkw"]
Oct 31 04:17:28.605555 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:28.605520    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bfbd25-939a-48b1-ae8e-e3cbf1313530" path="/var/lib/kubelet/pods/76bfbd25-939a-48b1-ae8e-e3cbf1313530/volumes"
Oct 31 04:17:30.491957 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.491923    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"]
Oct 31 04:17:30.492380 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.492119    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds" podUID="5bfa71e4-f250-4228-8e7d-426f43abffd5" containerName="maas-api" containerID="cri-o://1d8f83200b1552be3a85fbc25632b655c9134bd3de2ecd99c4ec70b75d0a2c17" gracePeriod=30
Oct 31 04:17:30.780562 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.780534    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:17:30.861138 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.861090    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfa71e4-f250-4228-8e7d-426f43abffd5-kube-api-access-5s94n\" (UniqueName: \"kubernetes.io/projected/5bfa71e4-f250-4228-8e7d-426f43abffd5-kube-api-access-5s94n\") pod \"5bfa71e4-f250-4228-8e7d-426f43abffd5\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") "
Oct 31 04:17:30.861325 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.861184    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-metrics-certs\") pod \"5bfa71e4-f250-4228-8e7d-426f43abffd5\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") "
Oct 31 04:17:30.861325 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.861221    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-maas-api-tls\") pod \"5bfa71e4-f250-4228-8e7d-426f43abffd5\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") "
Oct 31 04:17:30.861474 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.861405    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfa71e4-f250-4228-8e7d-426f43abffd5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfa71e4-f250-4228-8e7d-426f43abffd5-openshift-service-ca\") pod \"5bfa71e4-f250-4228-8e7d-426f43abffd5\" (UID: \"5bfa71e4-f250-4228-8e7d-426f43abffd5\") "
Oct 31 04:17:30.861813 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.861778    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfa71e4-f250-4228-8e7d-426f43abffd5-openshift-service-ca" pod "5bfa71e4-f250-4228-8e7d-426f43abffd5" (UID: "5bfa71e4-f250-4228-8e7d-426f43abffd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:17:30.864006 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.863971    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-maas-api-tls" pod "5bfa71e4-f250-4228-8e7d-426f43abffd5" (UID: "5bfa71e4-f250-4228-8e7d-426f43abffd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:17:30.864155 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.864043    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfa71e4-f250-4228-8e7d-426f43abffd5-kube-api-access-5s94n" pod "5bfa71e4-f250-4228-8e7d-426f43abffd5" (UID: "5bfa71e4-f250-4228-8e7d-426f43abffd5"). InnerVolumeSpecName "kube-api-access-5s94n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:17:30.864348 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.864316    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-metrics-certs" pod "5bfa71e4-f250-4228-8e7d-426f43abffd5" (UID: "5bfa71e4-f250-4228-8e7d-426f43abffd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:17:30.963272 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.963232    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s94n\" (UniqueName: \"kubernetes.io/projected/5bfa71e4-f250-4228-8e7d-426f43abffd5-kube-api-access-5s94n\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:30.963272 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.963266    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:30.963272 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.963276    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bfa71e4-f250-4228-8e7d-426f43abffd5-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:30.963514 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:30.963284    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfa71e4-f250-4228-8e7d-426f43abffd5-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:31.084489 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:31.084456    5415 generic.go:356] "Generic (PLEG): container finished" podID="5bfa71e4-f250-4228-8e7d-426f43abffd5" containerID="1d8f83200b1552be3a85fbc25632b655c9134bd3de2ecd99c4ec70b75d0a2c17" exitCode=0
Oct 31 04:17:31.084700 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:31.084507    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"
Oct 31 04:17:31.084700 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:31.084545    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds" event={"ID":"5bfa71e4-f250-4228-8e7d-426f43abffd5","Type":"ContainerDied","Data":"1d8f83200b1552be3a85fbc25632b655c9134bd3de2ecd99c4ec70b75d0a2c17"}
Oct 31 04:17:31.084700 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:31.084613    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds" event={"ID":"5bfa71e4-f250-4228-8e7d-426f43abffd5","Type":"ContainerDied","Data":"5fb6a23a218fd7c7ce10620028e5712c1739bcdfbcc83952e5416d5dea16a025"}
Oct 31 04:17:31.084700 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:31.084643    5415 scope.go:122] "RemoveContainer" containerID="1d8f83200b1552be3a85fbc25632b655c9134bd3de2ecd99c4ec70b75d0a2c17"
Oct 31 04:17:31.095971 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:31.095948    5415 scope.go:122] "RemoveContainer" containerID="1d8f83200b1552be3a85fbc25632b655c9134bd3de2ecd99c4ec70b75d0a2c17"
Oct 31 04:17:31.096297 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:17:31.096278    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8f83200b1552be3a85fbc25632b655c9134bd3de2ecd99c4ec70b75d0a2c17\": container with ID starting with 1d8f83200b1552be3a85fbc25632b655c9134bd3de2ecd99c4ec70b75d0a2c17 not found: ID does not exist" containerID="1d8f83200b1552be3a85fbc25632b655c9134bd3de2ecd99c4ec70b75d0a2c17"
Oct 31 04:17:31.096345 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:31.096307    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8f83200b1552be3a85fbc25632b655c9134bd3de2ecd99c4ec70b75d0a2c17"} err="failed to get container status \"1d8f83200b1552be3a85fbc25632b655c9134bd3de2ecd99c4ec70b75d0a2c17\": rpc error: code = NotFound desc = could not find container \"1d8f83200b1552be3a85fbc25632b655c9134bd3de2ecd99c4ec70b75d0a2c17\": container with ID starting with 1d8f83200b1552be3a85fbc25632b655c9134bd3de2ecd99c4ec70b75d0a2c17 not found: ID does not exist"
Oct 31 04:17:31.239683 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:31.239645    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"]
Oct 31 04:17:31.246131 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:31.246101    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2a54d-78796cdb8c-fzcds"]
Oct 31 04:17:32.606662 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:32.606627    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfa71e4-f250-4228-8e7d-426f43abffd5" path="/var/lib/kubelet/pods/5bfa71e4-f250-4228-8e7d-426f43abffd5/volumes"
Oct 31 04:17:46.176042 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:17:46.176004    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:17:59.601807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:59.601769    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"]
Oct 31 04:17:59.602406 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:59.602010    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl" podUID="782076e8-3b96-49a9-98f7-4134fc68c68b" containerName="maas-api" containerID="cri-o://352253814d226c54050a23c994bf59feffc86e86c8612090b327ba3c2704edb6" gracePeriod=30
Oct 31 04:17:59.889090 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:59.889058    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:17:59.992903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:59.992839    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-maas-api-tls\") pod \"782076e8-3b96-49a9-98f7-4134fc68c68b\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") "
Oct 31 04:17:59.993090 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:59.993023    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-metrics-certs\") pod \"782076e8-3b96-49a9-98f7-4134fc68c68b\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") "
Oct 31 04:17:59.993090 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:59.993050    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/782076e8-3b96-49a9-98f7-4134fc68c68b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/782076e8-3b96-49a9-98f7-4134fc68c68b-openshift-service-ca\") pod \"782076e8-3b96-49a9-98f7-4134fc68c68b\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") "
Oct 31 04:17:59.993218 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:59.993106    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782076e8-3b96-49a9-98f7-4134fc68c68b-kube-api-access-pldr4\" (UniqueName: \"kubernetes.io/projected/782076e8-3b96-49a9-98f7-4134fc68c68b-kube-api-access-pldr4\") pod \"782076e8-3b96-49a9-98f7-4134fc68c68b\" (UID: \"782076e8-3b96-49a9-98f7-4134fc68c68b\") "
Oct 31 04:17:59.993561 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:59.993528    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782076e8-3b96-49a9-98f7-4134fc68c68b-openshift-service-ca" pod "782076e8-3b96-49a9-98f7-4134fc68c68b" (UID: "782076e8-3b96-49a9-98f7-4134fc68c68b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:17:59.995238 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:59.995209    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-metrics-certs" pod "782076e8-3b96-49a9-98f7-4134fc68c68b" (UID: "782076e8-3b96-49a9-98f7-4134fc68c68b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:17:59.995373 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:59.995354    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782076e8-3b96-49a9-98f7-4134fc68c68b-kube-api-access-pldr4" pod "782076e8-3b96-49a9-98f7-4134fc68c68b" (UID: "782076e8-3b96-49a9-98f7-4134fc68c68b"). InnerVolumeSpecName "kube-api-access-pldr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:17:59.995499 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:17:59.995482    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-maas-api-tls" pod "782076e8-3b96-49a9-98f7-4134fc68c68b" (UID: "782076e8-3b96-49a9-98f7-4134fc68c68b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:18:00.094774 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.094732    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:00.094774 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.094767    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/782076e8-3b96-49a9-98f7-4134fc68c68b-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:00.094774 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.094780    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/782076e8-3b96-49a9-98f7-4134fc68c68b-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:00.095113 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.094791    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pldr4\" (UniqueName: \"kubernetes.io/projected/782076e8-3b96-49a9-98f7-4134fc68c68b-kube-api-access-pldr4\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:00.251417 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.251329    5415 generic.go:356] "Generic (PLEG): container finished" podID="782076e8-3b96-49a9-98f7-4134fc68c68b" containerID="352253814d226c54050a23c994bf59feffc86e86c8612090b327ba3c2704edb6" exitCode=0
Oct 31 04:18:00.251417 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.251379    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"
Oct 31 04:18:00.251623 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.251411    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl" event={"ID":"782076e8-3b96-49a9-98f7-4134fc68c68b","Type":"ContainerDied","Data":"352253814d226c54050a23c994bf59feffc86e86c8612090b327ba3c2704edb6"}
Oct 31 04:18:00.251623 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.251466    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl" event={"ID":"782076e8-3b96-49a9-98f7-4134fc68c68b","Type":"ContainerDied","Data":"a029d923ebdc729b6bbd31e608bd07f529128699496cfa571ec23222cec2f8a3"}
Oct 31 04:18:00.251623 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.251488    5415 scope.go:122] "RemoveContainer" containerID="352253814d226c54050a23c994bf59feffc86e86c8612090b327ba3c2704edb6"
Oct 31 04:18:00.262114 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.262094    5415 scope.go:122] "RemoveContainer" containerID="352253814d226c54050a23c994bf59feffc86e86c8612090b327ba3c2704edb6"
Oct 31 04:18:00.262428 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:18:00.262410    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352253814d226c54050a23c994bf59feffc86e86c8612090b327ba3c2704edb6\": container with ID starting with 352253814d226c54050a23c994bf59feffc86e86c8612090b327ba3c2704edb6 not found: ID does not exist" containerID="352253814d226c54050a23c994bf59feffc86e86c8612090b327ba3c2704edb6"
Oct 31 04:18:00.262493 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.262442    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352253814d226c54050a23c994bf59feffc86e86c8612090b327ba3c2704edb6"} err="failed to get container status \"352253814d226c54050a23c994bf59feffc86e86c8612090b327ba3c2704edb6\": rpc error: code = NotFound desc = could not find container \"352253814d226c54050a23c994bf59feffc86e86c8612090b327ba3c2704edb6\": container with ID starting with 352253814d226c54050a23c994bf59feffc86e86c8612090b327ba3c2704edb6 not found: ID does not exist"
Oct 31 04:18:00.411958 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.411919    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"]
Oct 31 04:18:00.416257 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.416226    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e53098-6fc456498b-d8msl"]
Oct 31 04:18:00.605682 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:00.605642    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782076e8-3b96-49a9-98f7-4134fc68c68b" path="/var/lib/kubelet/pods/782076e8-3b96-49a9-98f7-4134fc68c68b/volumes"
Oct 31 04:18:14.218247 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:14.218202    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"]
Oct 31 04:18:16.176869 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:18:16.176780    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:18:17.033472 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.033433    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"]
Oct 31 04:18:17.033929 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.033914    5415 cpu_manager.go:414] "containerMap: removing container" podUID="5bfa71e4-f250-4228-8e7d-426f43abffd5" containerName="maas-api"
Oct 31 04:18:17.033988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.033929    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfa71e4-f250-4228-8e7d-426f43abffd5" containerName="maas-api"
Oct 31 04:18:17.033988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.033943    5415 cpu_manager.go:414] "containerMap: removing container" podUID="6fa92af3-a192-4c80-bdb0-097b66fe9f14" containerName="cleanup"
Oct 31 04:18:17.033988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.033948    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa92af3-a192-4c80-bdb0-097b66fe9f14" containerName="cleanup"
Oct 31 04:18:17.033988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.033955    5415 cpu_manager.go:414] "containerMap: removing container" podUID="6a70215e-fc06-4236-91cb-bda11d72454b" containerName="cleanup"
Oct 31 04:18:17.033988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.033962    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a70215e-fc06-4236-91cb-bda11d72454b" containerName="cleanup"
Oct 31 04:18:17.033988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.033975    5415 cpu_manager.go:414] "containerMap: removing container" podUID="6fa92af3-a192-4c80-bdb0-097b66fe9f14" containerName="cleanup"
Oct 31 04:18:17.033988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.033980    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa92af3-a192-4c80-bdb0-097b66fe9f14" containerName="cleanup"
Oct 31 04:18:17.033988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.033986    5415 cpu_manager.go:414] "containerMap: removing container" podUID="83393f8e-2253-42a6-9fc0-25e5cc488155" containerName="istio-proxy"
Oct 31 04:18:17.033988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.033991    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83393f8e-2253-42a6-9fc0-25e5cc488155" containerName="istio-proxy"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.033996    5415 cpu_manager.go:414] "containerMap: removing container" podUID="0af401f3-6429-4711-bcaf-56c6bba19998" containerName="maas-api"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034001    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af401f3-6429-4711-bcaf-56c6bba19998" containerName="maas-api"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034010    5415 cpu_manager.go:414] "containerMap: removing container" podUID="5e4f1595-2c8a-4408-a037-9de9b29669cb" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034014    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4f1595-2c8a-4408-a037-9de9b29669cb" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034022    5415 cpu_manager.go:414] "containerMap: removing container" podUID="5e4f1595-2c8a-4408-a037-9de9b29669cb" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034027    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4f1595-2c8a-4408-a037-9de9b29669cb" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034037    5415 cpu_manager.go:414] "containerMap: removing container" podUID="d8034d48-df5b-40ae-a7a0-c026aad438ed" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034053    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8034d48-df5b-40ae-a7a0-c026aad438ed" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034059    5415 cpu_manager.go:414] "containerMap: removing container" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034064    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034071    5415 cpu_manager.go:414] "containerMap: removing container" podUID="d8034d48-df5b-40ae-a7a0-c026aad438ed" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034076    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8034d48-df5b-40ae-a7a0-c026aad438ed" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034089    5415 cpu_manager.go:414] "containerMap: removing container" podUID="96326289-4481-4634-b123-e93c01e0c1dc" containerName="istio-proxy"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034093    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96326289-4481-4634-b123-e93c01e0c1dc" containerName="istio-proxy"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034099    5415 cpu_manager.go:414] "containerMap: removing container" podUID="782076e8-3b96-49a9-98f7-4134fc68c68b" containerName="maas-api"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034103    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782076e8-3b96-49a9-98f7-4134fc68c68b" containerName="maas-api"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034110    5415 cpu_manager.go:414] "containerMap: removing container" podUID="b13e4466-9375-4740-9ba7-12d286d0142a" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034115    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13e4466-9375-4740-9ba7-12d286d0142a" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034127    5415 cpu_manager.go:414] "containerMap: removing container" podUID="76bfbd25-939a-48b1-ae8e-e3cbf1313530" containerName="payload-processing"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034131    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bfbd25-939a-48b1-ae8e-e3cbf1313530" containerName="payload-processing"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034138    5415 cpu_manager.go:414] "containerMap: removing container" podUID="b13e4466-9375-4740-9ba7-12d286d0142a" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034142    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13e4466-9375-4740-9ba7-12d286d0142a" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034148    5415 cpu_manager.go:414] "containerMap: removing container" podUID="b13e4466-9375-4740-9ba7-12d286d0142a" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034152    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13e4466-9375-4740-9ba7-12d286d0142a" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034161    5415 cpu_manager.go:414] "containerMap: removing container" podUID="d8034d48-df5b-40ae-a7a0-c026aad438ed" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034165    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8034d48-df5b-40ae-a7a0-c026aad438ed" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034172    5415 cpu_manager.go:414] "containerMap: removing container" podUID="6a70215e-fc06-4236-91cb-bda11d72454b" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034176    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a70215e-fc06-4236-91cb-bda11d72454b" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034184    5415 cpu_manager.go:414] "containerMap: removing container" podUID="6fa92af3-a192-4c80-bdb0-097b66fe9f14" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034188    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa92af3-a192-4c80-bdb0-097b66fe9f14" containerName="cleanup"
Oct 31 04:18:17.034260 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034271    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31e83da-c659-4596-ac00-c7e4369911bf" containerName="istio-proxy"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034283    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa92af3-a192-4c80-bdb0-097b66fe9f14" containerName="cleanup"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034289    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8034d48-df5b-40ae-a7a0-c026aad438ed" containerName="cleanup"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034298    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa92af3-a192-4c80-bdb0-097b66fe9f14" containerName="cleanup"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034303    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfa71e4-f250-4228-8e7d-426f43abffd5" containerName="maas-api"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034309    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a70215e-fc06-4236-91cb-bda11d72454b" containerName="cleanup"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034315    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="96326289-4481-4634-b123-e93c01e0c1dc" containerName="istio-proxy"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034320    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="782076e8-3b96-49a9-98f7-4134fc68c68b" containerName="maas-api"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034326    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="83393f8e-2253-42a6-9fc0-25e5cc488155" containerName="istio-proxy"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034331    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13e4466-9375-4740-9ba7-12d286d0142a" containerName="cleanup"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034338    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af401f3-6429-4711-bcaf-56c6bba19998" containerName="maas-api"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034344    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a70215e-fc06-4236-91cb-bda11d72454b" containerName="cleanup"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034351    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4f1595-2c8a-4408-a037-9de9b29669cb" containerName="cleanup"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034357    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bfbd25-939a-48b1-ae8e-e3cbf1313530" containerName="payload-processing"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034362    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4f1595-2c8a-4408-a037-9de9b29669cb" containerName="cleanup"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034369    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8034d48-df5b-40ae-a7a0-c026aad438ed" containerName="cleanup"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034374    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13e4466-9375-4740-9ba7-12d286d0142a" containerName="cleanup"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034379    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13e4466-9375-4740-9ba7-12d286d0142a" containerName="cleanup"
Oct 31 04:18:17.035186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.034384    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4f1595-2c8a-4408-a037-9de9b29669cb" containerName="cleanup"
Oct 31 04:18:17.037721 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.037697    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.080671 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.080633    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"]
Oct 31 04:18:17.081464 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.081430    5415 cpu_manager.go:414] "containerMap: removing container" podUID="5e4f1595-2c8a-4408-a037-9de9b29669cb" containerName="cleanup"
Oct 31 04:18:17.081464 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.081454    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4f1595-2c8a-4408-a037-9de9b29669cb" containerName="cleanup"
Oct 31 04:18:17.081684 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.081493    5415 cpu_manager.go:414] "containerMap: removing container" podUID="6a70215e-fc06-4236-91cb-bda11d72454b" containerName="cleanup"
Oct 31 04:18:17.081684 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.081499    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a70215e-fc06-4236-91cb-bda11d72454b" containerName="cleanup"
Oct 31 04:18:17.081684 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.081583    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a70215e-fc06-4236-91cb-bda11d72454b" containerName="cleanup"
Oct 31 04:18:17.081684 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.081594    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa92af3-a192-4c80-bdb0-097b66fe9f14" containerName="cleanup"
Oct 31 04:18:17.081684 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.081601    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8034d48-df5b-40ae-a7a0-c026aad438ed" containerName="cleanup"
Oct 31 04:18:17.084860 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.084840    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.085328 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.085300    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.085373 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.085362    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.085419 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.085390    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.085462 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.085425    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.085538 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.085519    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.085602 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.085589    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2l75\" (UniqueName: \"kubernetes.io/projected/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-kube-api-access-b2l75\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.085645 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.085633    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.085720 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.085706    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.085758 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.085736    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.096017 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.095985    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-96js4\""
Oct 31 04:18:17.141724 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.141688    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"]
Oct 31 04:18:17.186530 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.186457    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.187070 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.186641    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.187070 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.186795    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.187070 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.186849    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw4lc\" (UniqueName: \"kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-kube-api-access-hw4lc\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.187070 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.186910    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.187070 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.186948    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.187070 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.186955    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.187070 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.186975    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.187070 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.187029    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.187536 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.187086    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2l75\" (UniqueName: \"kubernetes.io/projected/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-kube-api-access-b2l75\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.187536 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.187137    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.187536 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.187164    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.187536 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.187209    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.187536 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.187252    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.187536 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.187289    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.187536 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.187331    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.187536 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.187348    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.187536 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.187378    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.187536 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.187421    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.188063 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.187728    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.188063 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.187804    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.188341 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.188315    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.188533 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.188509    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.191636 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.191604    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.192145 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.192124    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.193027 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.193008    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"]
Oct 31 04:18:17.242236 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.242199    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.242426 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.242291    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2l75\" (UniqueName: \"kubernetes.io/projected/c9d3bfd0-545c-4be0-92ae-30a7c68138ca-kube-api-access-b2l75\") pod \"maas-default-gateway-openshift-default-6886b96f9-bwzrc\" (UID: \"c9d3bfd0-545c-4be0-92ae-30a7c68138ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.288522 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.288418    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.288701 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.288585    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw4lc\" (UniqueName: \"kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-kube-api-access-hw4lc\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.288701 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.288624    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.288818 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.288710    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.288818 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.288805    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.288947 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.288896    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.288999 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.288961    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.289053 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.289000    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.289112 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.289059    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.289112 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.289086    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.289233 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.289128    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.289315 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.289288    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.289382 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.289332    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.289490 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.289469    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.291332 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.291306    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.291579 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.291559    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.342702 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.342669    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.342868 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.342808    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw4lc\" (UniqueName: \"kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-kube-api-access-hw4lc\") pod \"maas-default-gateway-openshift-default-6886b96f9-bgjqj\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.352046 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.352016    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:17.400102 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.400062    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:17.780801 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.780738    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"]
Oct 31 04:18:17.784114 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:18:17.784080    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d3bfd0_545c_4be0_92ae_30a7c68138ca.slice/crio-2bd33cd747a36995b0876a0e2161dc4dcdef9d995ef0675579abb44064f9c603 WatchSource:0}: Error finding container 2bd33cd747a36995b0876a0e2161dc4dcdef9d995ef0675579abb44064f9c603: Status 404 returned error can't find the container with id 2bd33cd747a36995b0876a0e2161dc4dcdef9d995ef0675579abb44064f9c603
Oct 31 04:18:17.786723 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.786689    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:18:17.786831 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.786759    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:18:17.786831 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.786790    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:18:17.829580 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.829546    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"]
Oct 31 04:18:17.833255 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:18:17.833226    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a09ce1_e619_406a_8097_ea1c5b8cbd3e.slice/crio-db7172fe7917659ffec685f53b76bd3e01da33c2d2ce7fb36bf72256e8dbee45 WatchSource:0}: Error finding container db7172fe7917659ffec685f53b76bd3e01da33c2d2ce7fb36bf72256e8dbee45: Status 404 returned error can't find the container with id db7172fe7917659ffec685f53b76bd3e01da33c2d2ce7fb36bf72256e8dbee45
Oct 31 04:18:17.837096 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.837052    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:18:17.837186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.837145    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:18:17.837242 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:17.837215    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:18:18.362746 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:18.362700    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc" event={"ID":"c9d3bfd0-545c-4be0-92ae-30a7c68138ca","Type":"ContainerStarted","Data":"6ded6678f165e7dceed54cf0bfe69b10cc9809c249b6d2833379eab27d5283ce"}
Oct 31 04:18:18.363268 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:18.362769    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc" event={"ID":"c9d3bfd0-545c-4be0-92ae-30a7c68138ca","Type":"ContainerStarted","Data":"2bd33cd747a36995b0876a0e2161dc4dcdef9d995ef0675579abb44064f9c603"}
Oct 31 04:18:18.364660 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:18.364618    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" event={"ID":"99a09ce1-e619-406a-8097-ea1c5b8cbd3e","Type":"ContainerStarted","Data":"f0683081c625e0a191ccff9c6ca658d2e1883a12a6627ee09ba1d79794c7421f"}
Oct 31 04:18:18.364802 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:18.364675    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" event={"ID":"99a09ce1-e619-406a-8097-ea1c5b8cbd3e","Type":"ContainerStarted","Data":"db7172fe7917659ffec685f53b76bd3e01da33c2d2ce7fb36bf72256e8dbee45"}
Oct 31 04:18:18.404936 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:18.402099    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:18.404936 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:18.403810    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bgjqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 31 04:18:18.404936 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:18.403871    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" podUID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 31 04:18:18.598901 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:18.593250    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc" podStartSLOduration=2.593228892 podStartE2EDuration="2.593228892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:18.477753576 +0000 UTC m=+4240.038233329" watchObservedRunningTime="2026-10-31 04:18:18.593228892 +0000 UTC m=+4240.153708621"
Oct 31 04:18:19.219262 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.219218    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t" podUID="321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2" containerName="payload-pre-processing" containerID="cri-o://e603f5f8cf3b2684c1b2f1cb2065c551edbdb2573834e87153d89d74258bea77" gracePeriod=25
Oct 31 04:18:19.362903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.355316    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:19.389905 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.379033    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bwzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:18:19.389905 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.379106    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc" podUID="c9d3bfd0-545c-4be0-92ae-30a7c68138ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 31 04:18:19.401834 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.401005    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bgjqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 31 04:18:19.401834 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.401070    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" podUID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 31 04:18:19.403908 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.403855    5415 generic.go:356] "Generic (PLEG): container finished" podID="321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2" containerID="e603f5f8cf3b2684c1b2f1cb2065c551edbdb2573834e87153d89d74258bea77" exitCode=0
Oct 31 04:18:19.404141 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.404111    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t" event={"ID":"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2","Type":"ContainerDied","Data":"e603f5f8cf3b2684c1b2f1cb2065c551edbdb2573834e87153d89d74258bea77"}
Oct 31 04:18:19.640736 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.640707    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"
Oct 31 04:18:19.718943 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.718845    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-kube-api-access-lrrmk\" (UniqueName: \"kubernetes.io/projected/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-kube-api-access-lrrmk\") pod \"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2\" (UID: \"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2\") "
Oct 31 04:18:19.718943 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.718944    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-plugins-config-volume\") pod \"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2\" (UID: \"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2\") "
Oct 31 04:18:19.719346 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.719321    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-plugins-config-volume" pod "321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2" (UID: "321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:18:19.721189 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.721164    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-kube-api-access-lrrmk" pod "321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2" (UID: "321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2"). InnerVolumeSpecName "kube-api-access-lrrmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:18:19.765917 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.757351    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" podStartSLOduration=2.757331079 podStartE2EDuration="2.757331079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:18.595122239 +0000 UTC m=+4240.155601982" watchObservedRunningTime="2026-10-31 04:18:19.757331079 +0000 UTC m=+4241.317810811"
Oct 31 04:18:19.820952 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.819803    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrrmk\" (UniqueName: \"kubernetes.io/projected/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-kube-api-access-lrrmk\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:19.820952 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:19.819838    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:20.356904 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:20.356506    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bwzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:18:20.356904 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:20.356594    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc" podUID="c9d3bfd0-545c-4be0-92ae-30a7c68138ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 31 04:18:20.401898 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:20.401440    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bgjqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 31 04:18:20.401898 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:20.401508    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" podUID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 31 04:18:20.412073 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:20.412044    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"
Oct 31 04:18:20.413336 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:20.413305    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t" event={"ID":"321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2","Type":"ContainerDied","Data":"501001b9ee262519f795f1130634d38cacfa0929a968651c1effd6403e3272e3"}
Oct 31 04:18:20.413431 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:20.413417    5415 scope.go:122] "RemoveContainer" containerID="e603f5f8cf3b2684c1b2f1cb2065c551edbdb2573834e87153d89d74258bea77"
Oct 31 04:18:20.580902 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:20.580682    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"]
Oct 31 04:18:20.625689 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:20.625604    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e679fe-795c685c9f-kzn8t"]
Oct 31 04:18:21.358903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:21.355344    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bwzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:18:21.358903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:21.355410    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc" podUID="c9d3bfd0-545c-4be0-92ae-30a7c68138ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 31 04:18:21.401006 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:21.400967    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bgjqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 31 04:18:21.401199 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:21.401032    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" podUID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 31 04:18:22.358903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:22.356368    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bwzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:18:22.358903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:22.356434    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc" podUID="c9d3bfd0-545c-4be0-92ae-30a7c68138ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 31 04:18:22.400587 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:22.400545    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bgjqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 31 04:18:22.400774 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:22.400616    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" podUID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 31 04:18:22.611098 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:22.611014    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2" path="/var/lib/kubelet/pods/321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2/volumes"
Oct 31 04:18:23.362911 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:23.359411    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bwzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:18:23.362911 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:23.359482    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc" podUID="c9d3bfd0-545c-4be0-92ae-30a7c68138ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 31 04:18:23.400561 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:23.400520    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bgjqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 31 04:18:23.400750 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:23.400602    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" podUID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 31 04:18:24.352538 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:24.352485    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bwzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:18:24.352761 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:24.352558    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc" podUID="c9d3bfd0-545c-4be0-92ae-30a7c68138ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 31 04:18:24.400817 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:24.400770    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bgjqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 31 04:18:24.401222 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:24.400843    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" podUID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 31 04:18:25.353230 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:25.353192    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bwzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:18:25.353440 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:25.353251    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc" podUID="c9d3bfd0-545c-4be0-92ae-30a7c68138ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 31 04:18:25.400780 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:25.400733    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bgjqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 31 04:18:25.400980 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:25.400809    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" podUID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 31 04:18:26.517124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:26.517091    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:26.517864 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:26.517534    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:26.519062 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:26.519043    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc"
Oct 31 04:18:26.568388 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:26.568355    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:26.569464 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:26.569405    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:26.570136 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:26.570116    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:18:32.148330 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.148285    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"]
Oct 31 04:18:32.149012 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.148990    5415 cpu_manager.go:414] "containerMap: removing container" podUID="321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2" containerName="payload-pre-processing"
Oct 31 04:18:32.149122 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.149013    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2" containerName="payload-pre-processing"
Oct 31 04:18:32.149215 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.149201    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="321cdce7-96ef-4f96-83b3-2f6a2ba1c6d2" containerName="payload-pre-processing"
Oct 31 04:18:32.153782 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.153756    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.258410 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.258371    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"]
Oct 31 04:18:32.307513 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.307472    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.307513 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.307520    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.307743 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.307587    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.307743 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.307611    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.307743 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.307638    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.307743 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.307700    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.307891 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.307847    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6fcf\" (UniqueName: \"kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-kube-api-access-s6fcf\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.307929 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.307904    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.307995 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.307974    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.409385 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.409279    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.409385 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.409330    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.409385 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.409356    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.409385 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.409379    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.409836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.409403    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.409836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.409436    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.409836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.409539    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6fcf\" (UniqueName: \"kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-kube-api-access-s6fcf\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.409836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.409568    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.409836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.409590    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.409836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.409824    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.410106 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.409858    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.410106 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.409954    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.410106 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.409989    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.410426 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.410406    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.412032 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.412016    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.412098 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.412082    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.464194 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.464150    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.464373 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.464321    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6fcf\" (UniqueName: \"kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-kube-api-access-s6fcf\") pod \"maas-default-gateway-openshift-default-6886b96f9-npjlx\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.468095 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.468072    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:32.888013 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.887978    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"]
Oct 31 04:18:32.926113 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:18:32.926072    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013cbda1_7eb1_4fbe_8e03_c2de7b71928a.slice/crio-e6db6a550c9e7c20d9e4e91a35fbd6e01a679757f4204b4ad9db15d28431cc57 WatchSource:0}: Error finding container e6db6a550c9e7c20d9e4e91a35fbd6e01a679757f4204b4ad9db15d28431cc57: Status 404 returned error can't find the container with id e6db6a550c9e7c20d9e4e91a35fbd6e01a679757f4204b4ad9db15d28431cc57
Oct 31 04:18:32.928836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.928805    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:18:32.928939 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.928918    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:18:32.928979 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:32.928950    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:18:33.497039 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:33.497000    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx" event={"ID":"013cbda1-7eb1-4fbe-8e03-c2de7b71928a","Type":"ContainerStarted","Data":"32b3117e60307f34566ea4fe20aea8a7f355014fa676c5f68f6e241ced695626"}
Oct 31 04:18:33.497498 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:33.497063    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx" event={"ID":"013cbda1-7eb1-4fbe-8e03-c2de7b71928a","Type":"ContainerStarted","Data":"e6db6a550c9e7c20d9e4e91a35fbd6e01a679757f4204b4ad9db15d28431cc57"}
Oct 31 04:18:33.615935 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:33.615849    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx" podStartSLOduration=1.6158293019999999 podStartE2EDuration="1.615829302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:33.615444391 +0000 UTC m=+4255.175924122" watchObservedRunningTime="2026-10-31 04:18:33.615829302 +0000 UTC m=+4255.176309034"
Oct 31 04:18:34.472688 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:34.472644    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:34.474614 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:34.474583    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-npjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 04:18:34.474771 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:34.474641    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx" podUID="013cbda1-7eb1-4fbe-8e03-c2de7b71928a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 04:18:35.472517 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:35.472454    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-npjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 04:18:35.473006 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:35.472543    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx" podUID="013cbda1-7eb1-4fbe-8e03-c2de7b71928a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 04:18:36.472169 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:36.472110    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-npjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 04:18:36.472381 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:36.472213    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx" podUID="013cbda1-7eb1-4fbe-8e03-c2de7b71928a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 04:18:37.472410 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:37.472367    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-npjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 04:18:37.472976 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:37.472462    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx" podUID="013cbda1-7eb1-4fbe-8e03-c2de7b71928a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 04:18:38.472062 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:38.472013    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-npjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 04:18:38.472283 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:38.472084    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx" podUID="013cbda1-7eb1-4fbe-8e03-c2de7b71928a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 04:18:39.469305 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:39.469267    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-npjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 04:18:39.469728 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:39.469357    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx" podUID="013cbda1-7eb1-4fbe-8e03-c2de7b71928a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 04:18:40.469346 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:40.469301    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-npjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 04:18:40.469892 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:40.469386    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx" podUID="013cbda1-7eb1-4fbe-8e03-c2de7b71928a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 04:18:41.662617 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:41.662577    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:41.663670 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:41.663614    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:41.664566 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:18:41.664548    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:18:46.176466 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:18:46.176423    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:19:16.176207 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:19:16.176168    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:19:46.176084 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:19:46.175999    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:20:16.175288 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:20:16.175244    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:20:46.175608 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:20:46.175565    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:21:16.177744 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:21:16.177663    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:21:16.753266 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:16.753224    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"]
Oct 31 04:21:16.753580 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:16.753511    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs" podUID="d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" containerName="istio-proxy" containerID="cri-o://9e34f15668c40bdcea4703d6abe1d6984ff1795cfa7f06df0e4a568acfa89cb5" gracePeriod=30
Oct 31 04:21:22.025636 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.025569    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 04:21:22.081010 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.080956    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-token\" (UniqueName: \"kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-token\") pod \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") "
Oct 31 04:21:22.081195 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.081018    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istiod-ca-cert\") pod \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") "
Oct 31 04:21:22.081195 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.081063    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-credential-socket\") pod \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") "
Oct 31 04:21:22.081195 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.081113    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-data\") pod \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") "
Oct 31 04:21:22.081339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.081204    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-podinfo\") pod \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") "
Oct 31 04:21:22.081339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.081242    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-kube-api-access-88nbt\" (UniqueName: \"kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-kube-api-access-88nbt\") pod \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") "
Oct 31 04:21:22.081339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.081280    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-envoy\") pod \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") "
Oct 31 04:21:22.081339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.081309    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-certs\") pod \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") "
Oct 31 04:21:22.081557 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.081425    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-credential-socket" pod "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" (UID: "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:21:22.081557 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.081451    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-data" pod "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" (UID: "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:21:22.081557 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.081488    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-socket\") pod \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\" (UID: \"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1\") "
Oct 31 04:21:22.081557 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.081510    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istiod-ca-cert" pod "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" (UID: "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:21:22.081785 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.081727    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-socket" pod "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" (UID: "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:21:22.081961 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.081930    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-certs" pod "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" (UID: "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:21:22.082141 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.082060    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:22.082141 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.082079    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:22.082141 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.082093    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:22.082141 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.082109    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:22.082141 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.082123    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:22.083491 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.083463    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-token" pod "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" (UID: "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:21:22.083593 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.083498    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-kube-api-access-88nbt" pod "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" (UID: "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1"). InnerVolumeSpecName "kube-api-access-88nbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:21:22.083827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.083800    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-envoy" pod "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" (UID: "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:21:22.083953 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.083800    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-podinfo" pod "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" (UID: "d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:21:22.182872 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.182838    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:22.182872 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.182866    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:22.183129 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.182910    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88nbt\" (UniqueName: \"kubernetes.io/projected/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-kube-api-access-88nbt\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:22.183129 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.182923    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:22.489132 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.489089    5415 generic.go:356] "Generic (PLEG): container finished" podID="d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" containerID="9e34f15668c40bdcea4703d6abe1d6984ff1795cfa7f06df0e4a568acfa89cb5" exitCode=0
Oct 31 04:21:22.489323 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.489151    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"
Oct 31 04:21:22.489323 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.489145    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs" event={"ID":"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1","Type":"ContainerDied","Data":"9e34f15668c40bdcea4703d6abe1d6984ff1795cfa7f06df0e4a568acfa89cb5"}
Oct 31 04:21:22.489323 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.489281    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs" event={"ID":"d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1","Type":"ContainerDied","Data":"d0a88937c3dc85192f8bbae8b747e371a8cea8289194930e83150b59172cfbd9"}
Oct 31 04:21:22.489323 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.489293    5415 scope.go:122] "RemoveContainer" containerID="9e34f15668c40bdcea4703d6abe1d6984ff1795cfa7f06df0e4a568acfa89cb5"
Oct 31 04:21:22.500207 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.500186    5415 scope.go:122] "RemoveContainer" containerID="9e34f15668c40bdcea4703d6abe1d6984ff1795cfa7f06df0e4a568acfa89cb5"
Oct 31 04:21:22.500495 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:21:22.500472    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e34f15668c40bdcea4703d6abe1d6984ff1795cfa7f06df0e4a568acfa89cb5\": container with ID starting with 9e34f15668c40bdcea4703d6abe1d6984ff1795cfa7f06df0e4a568acfa89cb5 not found: ID does not exist" containerID="9e34f15668c40bdcea4703d6abe1d6984ff1795cfa7f06df0e4a568acfa89cb5"
Oct 31 04:21:22.500573 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.500504    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e34f15668c40bdcea4703d6abe1d6984ff1795cfa7f06df0e4a568acfa89cb5"} err="failed to get container status \"9e34f15668c40bdcea4703d6abe1d6984ff1795cfa7f06df0e4a568acfa89cb5\": rpc error: code = NotFound desc = could not find container \"9e34f15668c40bdcea4703d6abe1d6984ff1795cfa7f06df0e4a568acfa89cb5\": container with ID starting with 9e34f15668c40bdcea4703d6abe1d6984ff1795cfa7f06df0e4a568acfa89cb5 not found: ID does not exist"
Oct 31 04:21:22.645623 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.645590    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"]
Oct 31 04:21:22.659775 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:22.659738    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-ksczs"]
Oct 31 04:21:24.605390 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:21:24.605349    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" path="/var/lib/kubelet/pods/d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1/volumes"
Oct 31 04:21:46.175530 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:21:46.175491    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:22:16.176022 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:22:16.175984    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:22:38.963136 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:38.963097    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"]
Oct 31 04:22:39.010525 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:39.010478    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"]
Oct 31 04:22:39.012716 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:39.012686    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"]
Oct 31 04:22:39.016996 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:39.016968    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"]
Oct 31 04:22:46.175983 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:22:46.175868    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:22:53.984152 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:53.984103    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq" podUID="37142a22-1f4a-4bbe-ac5b-72a3ff276832" containerName="main" containerID="cri-o://04f60ce4e7af25a76872ae3f7445c36f1b52bfb642352ddae9ac2aece396db1f" gracePeriod=45
Oct 31 04:22:54.032300 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.032251    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl" podUID="187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51" containerName="main" containerID="cri-o://055c65178450b786580a01920c713df5fbc12a8aad7463d0b20e258a937d16b5" gracePeriod=45
Oct 31 04:22:54.033989 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.033936    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" podUID="7286f18c-2456-47f7-a04d-078677b37a27" containerName="main" containerID="cri-o://fbdba167ed831140cb7d8c45547e32e26b74ed5ffeb32496f2b681d3ddda5541" gracePeriod=45
Oct 31 04:22:54.037499 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.037462    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd" podUID="e8a80930-06e4-4b4e-9aec-79274f0961f0" containerName="main" containerID="cri-o://cd9970a7d7f35d370af07c7701c8d0f1ec967fc7d2128f118753da8c0b15be45" gracePeriod=45
Oct 31 04:22:54.322132 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.321753    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 04:22:54.353014 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.352970    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 04:22:54.365230 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.365200    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 04:22:54.371389 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.371361    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 04:22:54.374330 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.374301    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tmp-dir\") pod \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") "
Oct 31 04:22:54.374486 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.374376    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-kube-api-access-bgw8v\" (UniqueName: \"kubernetes.io/projected/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-kube-api-access-bgw8v\") pod \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") "
Oct 31 04:22:54.374486 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.374423    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-model-cache\") pod \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") "
Oct 31 04:22:54.374611 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.374571    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tmp-dir" pod "187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51" (UID: "187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.374611 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.374585    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tls-certs\" (UniqueName: \"kubernetes.io/secret/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tls-certs\") pod \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") "
Oct 31 04:22:54.374721 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.374613    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-dshm\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-dshm\") pod \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") "
Oct 31 04:22:54.374721 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.374641    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-home\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-home\") pod \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\" (UID: \"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51\") "
Oct 31 04:22:54.374823 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.374749    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-model-cache" pod "187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51" (UID: "187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.375066 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.375037    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-home" pod "187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51" (UID: "187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.375194 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.375096    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.375194 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.375113    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.377196 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.377169    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-kube-api-access-bgw8v" pod "187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51" (UID: "187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51"). InnerVolumeSpecName "kube-api-access-bgw8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:22:54.377319 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.377242    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-dshm" pod "187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51" (UID: "187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.377365 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.377352    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tls-certs" pod "187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51" (UID: "187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:22:54.476065 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476016    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-tmp-dir\") pod \"7286f18c-2456-47f7-a04d-078677b37a27\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") "
Oct 31 04:22:54.476273 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476090    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8a80930-06e4-4b4e-9aec-79274f0961f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a80930-06e4-4b4e-9aec-79274f0961f0-tls-certs\") pod \"e8a80930-06e4-4b4e-9aec-79274f0961f0\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") "
Oct 31 04:22:54.476273 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476121    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7286f18c-2456-47f7-a04d-078677b37a27-kube-api-access-8w5lr\" (UniqueName: \"kubernetes.io/projected/7286f18c-2456-47f7-a04d-078677b37a27-kube-api-access-8w5lr\") pod \"7286f18c-2456-47f7-a04d-078677b37a27\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") "
Oct 31 04:22:54.476273 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476196    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-tmp-dir\") pod \"e8a80930-06e4-4b4e-9aec-79274f0961f0\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") "
Oct 31 04:22:54.476273 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476225    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-home\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-home\") pod \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") "
Oct 31 04:22:54.476273 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476252    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37142a22-1f4a-4bbe-ac5b-72a3ff276832-kube-api-access-dh9ph\" (UniqueName: \"kubernetes.io/projected/37142a22-1f4a-4bbe-ac5b-72a3ff276832-kube-api-access-dh9ph\") pod \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") "
Oct 31 04:22:54.476541 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476284    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tls-certs\" (UniqueName: \"kubernetes.io/secret/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tls-certs\") pod \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") "
Oct 31 04:22:54.476541 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476313    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7286f18c-2456-47f7-a04d-078677b37a27-tls-certs\" (UniqueName: \"kubernetes.io/secret/7286f18c-2456-47f7-a04d-078677b37a27-tls-certs\") pod \"7286f18c-2456-47f7-a04d-078677b37a27\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") "
Oct 31 04:22:54.476541 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476312    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-tmp-dir" pod "7286f18c-2456-47f7-a04d-078677b37a27" (UID: "7286f18c-2456-47f7-a04d-078677b37a27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.476541 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476353    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-dshm\") pod \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") "
Oct 31 04:22:54.476541 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476382    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a80930-06e4-4b4e-9aec-79274f0961f0-kube-api-access-78mzh\" (UniqueName: \"kubernetes.io/projected/e8a80930-06e4-4b4e-9aec-79274f0961f0-kube-api-access-78mzh\") pod \"e8a80930-06e4-4b4e-9aec-79274f0961f0\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") "
Oct 31 04:22:54.476541 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476434    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-dshm\") pod \"7286f18c-2456-47f7-a04d-078677b37a27\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") "
Oct 31 04:22:54.476541 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476490    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-model-cache\") pod \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") "
Oct 31 04:22:54.476950 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476564    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-home\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-home\") pod \"e8a80930-06e4-4b4e-9aec-79274f0961f0\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") "
Oct 31 04:22:54.476950 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476614    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-model-cache\") pod \"e8a80930-06e4-4b4e-9aec-79274f0961f0\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") "
Oct 31 04:22:54.476950 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476644    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-dshm\") pod \"e8a80930-06e4-4b4e-9aec-79274f0961f0\" (UID: \"e8a80930-06e4-4b4e-9aec-79274f0961f0\") "
Oct 31 04:22:54.476950 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476672    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-model-cache\") pod \"7286f18c-2456-47f7-a04d-078677b37a27\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") "
Oct 31 04:22:54.476950 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476732    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tmp-dir\") pod \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\" (UID: \"37142a22-1f4a-4bbe-ac5b-72a3ff276832\") "
Oct 31 04:22:54.476950 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.476832    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-home\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-home\") pod \"7286f18c-2456-47f7-a04d-078677b37a27\" (UID: \"7286f18c-2456-47f7-a04d-078677b37a27\") "
Oct 31 04:22:54.477283 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.477231    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-model-cache" pod "37142a22-1f4a-4bbe-ac5b-72a3ff276832" (UID: "37142a22-1f4a-4bbe-ac5b-72a3ff276832"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.477499 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.477473    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-home" pod "7286f18c-2456-47f7-a04d-078677b37a27" (UID: "7286f18c-2456-47f7-a04d-078677b37a27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.477652 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.477621    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-home" pod "e8a80930-06e4-4b4e-9aec-79274f0961f0" (UID: "e8a80930-06e4-4b4e-9aec-79274f0961f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.478216 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.477674    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-model-cache" pod "7286f18c-2456-47f7-a04d-078677b37a27" (UID: "7286f18c-2456-47f7-a04d-078677b37a27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.478216 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.477824    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-tmp-dir" pod "e8a80930-06e4-4b4e-9aec-79274f0961f0" (UID: "e8a80930-06e4-4b4e-9aec-79274f0961f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.478216 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.477856    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tmp-dir" pod "37142a22-1f4a-4bbe-ac5b-72a3ff276832" (UID: "37142a22-1f4a-4bbe-ac5b-72a3ff276832"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.478216 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.478070    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-model-cache" pod "e8a80930-06e4-4b4e-9aec-79274f0961f0" (UID: "e8a80930-06e4-4b4e-9aec-79274f0961f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.478613 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.478293    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-home" pod "37142a22-1f4a-4bbe-ac5b-72a3ff276832" (UID: "37142a22-1f4a-4bbe-ac5b-72a3ff276832"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.480043 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.479323    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.480043 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.479370    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.480043 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.479387    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgw8v\" (UniqueName: \"kubernetes.io/projected/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-kube-api-access-bgw8v\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.480043 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.479405    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.480043 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.479522    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.480043 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.479540    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.480043 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.479674    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a80930-06e4-4b4e-9aec-79274f0961f0-tls-certs" pod "e8a80930-06e4-4b4e-9aec-79274f0961f0" (UID: "e8a80930-06e4-4b4e-9aec-79274f0961f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:22:54.481532 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.481500    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37142a22-1f4a-4bbe-ac5b-72a3ff276832-kube-api-access-dh9ph" pod "37142a22-1f4a-4bbe-ac5b-72a3ff276832" (UID: "37142a22-1f4a-4bbe-ac5b-72a3ff276832"). InnerVolumeSpecName "kube-api-access-dh9ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:22:54.482454 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.482426    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7286f18c-2456-47f7-a04d-078677b37a27-tls-certs" pod "7286f18c-2456-47f7-a04d-078677b37a27" (UID: "7286f18c-2456-47f7-a04d-078677b37a27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:22:54.482676 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.482653    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7286f18c-2456-47f7-a04d-078677b37a27-kube-api-access-8w5lr" pod "7286f18c-2456-47f7-a04d-078677b37a27" (UID: "7286f18c-2456-47f7-a04d-078677b37a27"). InnerVolumeSpecName "kube-api-access-8w5lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:22:54.483275 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.483234    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-dshm" pod "e8a80930-06e4-4b4e-9aec-79274f0961f0" (UID: "e8a80930-06e4-4b4e-9aec-79274f0961f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.484053 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.484029    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tls-certs" pod "37142a22-1f4a-4bbe-ac5b-72a3ff276832" (UID: "37142a22-1f4a-4bbe-ac5b-72a3ff276832"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:22:54.484257 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.484229    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-dshm" pod "37142a22-1f4a-4bbe-ac5b-72a3ff276832" (UID: "37142a22-1f4a-4bbe-ac5b-72a3ff276832"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.484621 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.484602    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a80930-06e4-4b4e-9aec-79274f0961f0-kube-api-access-78mzh" pod "e8a80930-06e4-4b4e-9aec-79274f0961f0" (UID: "e8a80930-06e4-4b4e-9aec-79274f0961f0"). InnerVolumeSpecName "kube-api-access-78mzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:22:54.485044 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.485014    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-dshm" pod "7286f18c-2456-47f7-a04d-078677b37a27" (UID: "7286f18c-2456-47f7-a04d-078677b37a27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:22:54.580394 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580296    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a80930-06e4-4b4e-9aec-79274f0961f0-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.580394 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580329    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w5lr\" (UniqueName: \"kubernetes.io/projected/7286f18c-2456-47f7-a04d-078677b37a27-kube-api-access-8w5lr\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.580394 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580342    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.580394 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580354    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.580394 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580364    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh9ph\" (UniqueName: \"kubernetes.io/projected/37142a22-1f4a-4bbe-ac5b-72a3ff276832-kube-api-access-dh9ph\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.580394 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580374    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.580394 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580383    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7286f18c-2456-47f7-a04d-078677b37a27-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.580394 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580392    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.580394 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580401    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78mzh\" (UniqueName: \"kubernetes.io/projected/e8a80930-06e4-4b4e-9aec-79274f0961f0-kube-api-access-78mzh\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.581064 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580410    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.581064 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580423    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.581064 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580433    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.581064 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580442    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a80930-06e4-4b4e-9aec-79274f0961f0-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.581064 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580450    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.581064 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580458    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37142a22-1f4a-4bbe-ac5b-72a3ff276832-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:54.581064 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:54.580466    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7286f18c-2456-47f7-a04d-078677b37a27-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:55.036973 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.036937    5415 generic.go:356] "Generic (PLEG): container finished" podID="37142a22-1f4a-4bbe-ac5b-72a3ff276832" containerID="04f60ce4e7af25a76872ae3f7445c36f1b52bfb642352ddae9ac2aece396db1f" exitCode=0
Oct 31 04:22:55.037422 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.036987    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"
Oct 31 04:22:55.037422 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.037043    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq" event={"ID":"37142a22-1f4a-4bbe-ac5b-72a3ff276832","Type":"ContainerDied","Data":"04f60ce4e7af25a76872ae3f7445c36f1b52bfb642352ddae9ac2aece396db1f"}
Oct 31 04:22:55.037422 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.037095    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq" event={"ID":"37142a22-1f4a-4bbe-ac5b-72a3ff276832","Type":"ContainerDied","Data":"9353fec719c65c7547cf54664a9e38392807f58af6fcccd6a43664b513e5f003"}
Oct 31 04:22:55.037422 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.037120    5415 scope.go:122] "RemoveContainer" containerID="04f60ce4e7af25a76872ae3f7445c36f1b52bfb642352ddae9ac2aece396db1f"
Oct 31 04:22:55.038827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.038805    5415 generic.go:356] "Generic (PLEG): container finished" podID="e8a80930-06e4-4b4e-9aec-79274f0961f0" containerID="cd9970a7d7f35d370af07c7701c8d0f1ec967fc7d2128f118753da8c0b15be45" exitCode=0
Oct 31 04:22:55.038963 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.038860    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"
Oct 31 04:22:55.038963 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.038923    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd" event={"ID":"e8a80930-06e4-4b4e-9aec-79274f0961f0","Type":"ContainerDied","Data":"cd9970a7d7f35d370af07c7701c8d0f1ec967fc7d2128f118753da8c0b15be45"}
Oct 31 04:22:55.038963 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.038959    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd" event={"ID":"e8a80930-06e4-4b4e-9aec-79274f0961f0","Type":"ContainerDied","Data":"e2698c08ef80db9bdef5ace5261253282821e469ce3c1f99dff06605f01214e8"}
Oct 31 04:22:55.040589 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.040565    5415 generic.go:356] "Generic (PLEG): container finished" podID="187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51" containerID="055c65178450b786580a01920c713df5fbc12a8aad7463d0b20e258a937d16b5" exitCode=0
Oct 31 04:22:55.040735 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.040608    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"
Oct 31 04:22:55.040735 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.040644    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl" event={"ID":"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51","Type":"ContainerDied","Data":"055c65178450b786580a01920c713df5fbc12a8aad7463d0b20e258a937d16b5"}
Oct 31 04:22:55.040735 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.040688    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl" event={"ID":"187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51","Type":"ContainerDied","Data":"a822223496e9d6027a3479ad4e986c75e9e7679e8f63c1199411197cfa4a9345"}
Oct 31 04:22:55.042467 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.042443    5415 generic.go:356] "Generic (PLEG): container finished" podID="7286f18c-2456-47f7-a04d-078677b37a27" containerID="fbdba167ed831140cb7d8c45547e32e26b74ed5ffeb32496f2b681d3ddda5541" exitCode=0
Oct 31 04:22:55.042550 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.042479    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"
Oct 31 04:22:55.042550 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.042480    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" event={"ID":"7286f18c-2456-47f7-a04d-078677b37a27","Type":"ContainerDied","Data":"fbdba167ed831140cb7d8c45547e32e26b74ed5ffeb32496f2b681d3ddda5541"}
Oct 31 04:22:55.042550 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.042513    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg" event={"ID":"7286f18c-2456-47f7-a04d-078677b37a27","Type":"ContainerDied","Data":"fab54138ae2ad1a919e5d019c6a1431c256e0083351438568ce0ee96436ff7f8"}
Oct 31 04:22:55.048336 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.048298    5415 scope.go:122] "RemoveContainer" containerID="04f60ce4e7af25a76872ae3f7445c36f1b52bfb642352ddae9ac2aece396db1f"
Oct 31 04:22:55.048638 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:22:55.048610    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f60ce4e7af25a76872ae3f7445c36f1b52bfb642352ddae9ac2aece396db1f\": container with ID starting with 04f60ce4e7af25a76872ae3f7445c36f1b52bfb642352ddae9ac2aece396db1f not found: ID does not exist" containerID="04f60ce4e7af25a76872ae3f7445c36f1b52bfb642352ddae9ac2aece396db1f"
Oct 31 04:22:55.048698 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.048655    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f60ce4e7af25a76872ae3f7445c36f1b52bfb642352ddae9ac2aece396db1f"} err="failed to get container status \"04f60ce4e7af25a76872ae3f7445c36f1b52bfb642352ddae9ac2aece396db1f\": rpc error: code = NotFound desc = could not find container \"04f60ce4e7af25a76872ae3f7445c36f1b52bfb642352ddae9ac2aece396db1f\": container with ID starting with 04f60ce4e7af25a76872ae3f7445c36f1b52bfb642352ddae9ac2aece396db1f not found: ID does not exist"
Oct 31 04:22:55.048698 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.048680    5415 scope.go:122] "RemoveContainer" containerID="cd9970a7d7f35d370af07c7701c8d0f1ec967fc7d2128f118753da8c0b15be45"
Oct 31 04:22:55.059276 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.059253    5415 scope.go:122] "RemoveContainer" containerID="cd9970a7d7f35d370af07c7701c8d0f1ec967fc7d2128f118753da8c0b15be45"
Oct 31 04:22:55.059594 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:22:55.059573    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9970a7d7f35d370af07c7701c8d0f1ec967fc7d2128f118753da8c0b15be45\": container with ID starting with cd9970a7d7f35d370af07c7701c8d0f1ec967fc7d2128f118753da8c0b15be45 not found: ID does not exist" containerID="cd9970a7d7f35d370af07c7701c8d0f1ec967fc7d2128f118753da8c0b15be45"
Oct 31 04:22:55.059655 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.059603    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9970a7d7f35d370af07c7701c8d0f1ec967fc7d2128f118753da8c0b15be45"} err="failed to get container status \"cd9970a7d7f35d370af07c7701c8d0f1ec967fc7d2128f118753da8c0b15be45\": rpc error: code = NotFound desc = could not find container \"cd9970a7d7f35d370af07c7701c8d0f1ec967fc7d2128f118753da8c0b15be45\": container with ID starting with cd9970a7d7f35d370af07c7701c8d0f1ec967fc7d2128f118753da8c0b15be45 not found: ID does not exist"
Oct 31 04:22:55.059655 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.059621    5415 scope.go:122] "RemoveContainer" containerID="055c65178450b786580a01920c713df5fbc12a8aad7463d0b20e258a937d16b5"
Oct 31 04:22:55.068453 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.068431    5415 scope.go:122] "RemoveContainer" containerID="055c65178450b786580a01920c713df5fbc12a8aad7463d0b20e258a937d16b5"
Oct 31 04:22:55.068767 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:22:55.068742    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055c65178450b786580a01920c713df5fbc12a8aad7463d0b20e258a937d16b5\": container with ID starting with 055c65178450b786580a01920c713df5fbc12a8aad7463d0b20e258a937d16b5 not found: ID does not exist" containerID="055c65178450b786580a01920c713df5fbc12a8aad7463d0b20e258a937d16b5"
Oct 31 04:22:55.068828 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.068778    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055c65178450b786580a01920c713df5fbc12a8aad7463d0b20e258a937d16b5"} err="failed to get container status \"055c65178450b786580a01920c713df5fbc12a8aad7463d0b20e258a937d16b5\": rpc error: code = NotFound desc = could not find container \"055c65178450b786580a01920c713df5fbc12a8aad7463d0b20e258a937d16b5\": container with ID starting with 055c65178450b786580a01920c713df5fbc12a8aad7463d0b20e258a937d16b5 not found: ID does not exist"
Oct 31 04:22:55.068828 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.068795    5415 scope.go:122] "RemoveContainer" containerID="fbdba167ed831140cb7d8c45547e32e26b74ed5ffeb32496f2b681d3ddda5541"
Oct 31 04:22:55.078051 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.078030    5415 scope.go:122] "RemoveContainer" containerID="fbdba167ed831140cb7d8c45547e32e26b74ed5ffeb32496f2b681d3ddda5541"
Oct 31 04:22:55.078373 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:22:55.078353    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbdba167ed831140cb7d8c45547e32e26b74ed5ffeb32496f2b681d3ddda5541\": container with ID starting with fbdba167ed831140cb7d8c45547e32e26b74ed5ffeb32496f2b681d3ddda5541 not found: ID does not exist" containerID="fbdba167ed831140cb7d8c45547e32e26b74ed5ffeb32496f2b681d3ddda5541"
Oct 31 04:22:55.078440 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.078384    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdba167ed831140cb7d8c45547e32e26b74ed5ffeb32496f2b681d3ddda5541"} err="failed to get container status \"fbdba167ed831140cb7d8c45547e32e26b74ed5ffeb32496f2b681d3ddda5541\": rpc error: code = NotFound desc = could not find container \"fbdba167ed831140cb7d8c45547e32e26b74ed5ffeb32496f2b681d3ddda5541\": container with ID starting with fbdba167ed831140cb7d8c45547e32e26b74ed5ffeb32496f2b681d3ddda5541 not found: ID does not exist"
Oct 31 04:22:55.191107 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.191067    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"]
Oct 31 04:22:55.197017 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.196986    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/simulator-w3-dd5b85-kserve-67c8d46dcd-bm8nd"]
Oct 31 04:22:55.256066 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.255971    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl" podUID="187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:22:55.300019 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.299981    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"]
Oct 31 04:22:55.303468 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.303438    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/distinct-2-w3-dd5b85-kserve-ddf45bc85-dzjfq"]
Oct 31 04:22:55.405911 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.405842    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"]
Oct 31 04:22:55.409537 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.409505    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/premium-w3-dd5b85-kserve-544fbc7799-bjqsg"]
Oct 31 04:22:55.512444 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.512353    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"]
Oct 31 04:22:55.514062 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:55.514032    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd5b85/distinct-w3-dd5b85-kserve-69c9d8b874-k92xl"]
Oct 31 04:22:56.605700 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:56.605668    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51" path="/var/lib/kubelet/pods/187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51/volumes"
Oct 31 04:22:56.606220 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:56.606203    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37142a22-1f4a-4bbe-ac5b-72a3ff276832" path="/var/lib/kubelet/pods/37142a22-1f4a-4bbe-ac5b-72a3ff276832/volumes"
Oct 31 04:22:56.606591 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:56.606577    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7286f18c-2456-47f7-a04d-078677b37a27" path="/var/lib/kubelet/pods/7286f18c-2456-47f7-a04d-078677b37a27/volumes"
Oct 31 04:22:56.607316 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:22:56.607292    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a80930-06e4-4b4e-9aec-79274f0961f0" path="/var/lib/kubelet/pods/e8a80930-06e4-4b4e-9aec-79274f0961f0/volumes"
Oct 31 04:23:16.176335 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:23:16.176295    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:23:46.175149 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:23:46.175109    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:24:16.176098 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:24:16.176006    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:24:46.175749 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:24:46.175708    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:25:16.176198 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:25:16.176155    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:25:46.175732 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:25:46.175638    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:26:11.711155 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:11.711105    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"]
Oct 31 04:26:16.174483 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:26:16.174445    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:26:16.711598 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:16.711529    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9" podUID="35e6897c-0910-4afe-bdbb-7a19250f6735" containerName="payload-pre-processing" containerID="cri-o://71ab76403b72bc308cf01f73ec8b898be077df644ad35cd2e422485b0dfd2626" gracePeriod=25
Oct 31 04:26:20.969268 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:20.969225    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9" podUID="35e6897c-0910-4afe-bdbb-7a19250f6735" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:26:30.969530 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:30.969487    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9" podUID="35e6897c-0910-4afe-bdbb-7a19250f6735" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:26:36.990074 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:36.990036    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"
Oct 31 04:26:37.107575 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.107546    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e6897c-0910-4afe-bdbb-7a19250f6735-kube-api-access-lk2zz\" (UniqueName: \"kubernetes.io/projected/35e6897c-0910-4afe-bdbb-7a19250f6735-kube-api-access-lk2zz\") pod \"35e6897c-0910-4afe-bdbb-7a19250f6735\" (UID: \"35e6897c-0910-4afe-bdbb-7a19250f6735\") "
Oct 31 04:26:37.107771 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.107582    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35e6897c-0910-4afe-bdbb-7a19250f6735-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e6897c-0910-4afe-bdbb-7a19250f6735-plugins-config-volume\") pod \"35e6897c-0910-4afe-bdbb-7a19250f6735\" (UID: \"35e6897c-0910-4afe-bdbb-7a19250f6735\") "
Oct 31 04:26:37.108033 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.108008    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e6897c-0910-4afe-bdbb-7a19250f6735-plugins-config-volume" pod "35e6897c-0910-4afe-bdbb-7a19250f6735" (UID: "35e6897c-0910-4afe-bdbb-7a19250f6735"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:26:37.110263 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.110238    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e6897c-0910-4afe-bdbb-7a19250f6735-kube-api-access-lk2zz" pod "35e6897c-0910-4afe-bdbb-7a19250f6735" (UID: "35e6897c-0910-4afe-bdbb-7a19250f6735"). InnerVolumeSpecName "kube-api-access-lk2zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:37.208967 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.208917    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk2zz\" (UniqueName: \"kubernetes.io/projected/35e6897c-0910-4afe-bdbb-7a19250f6735-kube-api-access-lk2zz\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:37.208967 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.208960    5415 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e6897c-0910-4afe-bdbb-7a19250f6735-plugins-config-volume\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:37.246788 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.246693    5415 generic.go:356] "Generic (PLEG): container finished" podID="35e6897c-0910-4afe-bdbb-7a19250f6735" containerID="71ab76403b72bc308cf01f73ec8b898be077df644ad35cd2e422485b0dfd2626" exitCode=0
Oct 31 04:26:37.246788 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.246743    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"
Oct 31 04:26:37.246788 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.246747    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9" event={"ID":"35e6897c-0910-4afe-bdbb-7a19250f6735","Type":"ContainerDied","Data":"71ab76403b72bc308cf01f73ec8b898be077df644ad35cd2e422485b0dfd2626"}
Oct 31 04:26:37.247103 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.246800    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9" event={"ID":"35e6897c-0910-4afe-bdbb-7a19250f6735","Type":"ContainerDied","Data":"fb63e7b298db79a16b5f6d6ddf942a6b3ec8f6a8c030a4edf1aec25556528a23"}
Oct 31 04:26:37.247103 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.246829    5415 scope.go:122] "RemoveContainer" containerID="71ab76403b72bc308cf01f73ec8b898be077df644ad35cd2e422485b0dfd2626"
Oct 31 04:26:37.256647 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.256625    5415 scope.go:122] "RemoveContainer" containerID="71ab76403b72bc308cf01f73ec8b898be077df644ad35cd2e422485b0dfd2626"
Oct 31 04:26:37.256993 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:26:37.256969    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ab76403b72bc308cf01f73ec8b898be077df644ad35cd2e422485b0dfd2626\": container with ID starting with 71ab76403b72bc308cf01f73ec8b898be077df644ad35cd2e422485b0dfd2626 not found: ID does not exist" containerID="71ab76403b72bc308cf01f73ec8b898be077df644ad35cd2e422485b0dfd2626"
Oct 31 04:26:37.257057 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.257003    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ab76403b72bc308cf01f73ec8b898be077df644ad35cd2e422485b0dfd2626"} err="failed to get container status \"71ab76403b72bc308cf01f73ec8b898be077df644ad35cd2e422485b0dfd2626\": rpc error: code = NotFound desc = could not find container \"71ab76403b72bc308cf01f73ec8b898be077df644ad35cd2e422485b0dfd2626\": container with ID starting with 71ab76403b72bc308cf01f73ec8b898be077df644ad35cd2e422485b0dfd2626 not found: ID does not exist"
Oct 31 04:26:37.402199 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.402155    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"]
Oct 31 04:26:37.404792 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:37.404763    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5f70c-66845bbcc5-cwrh9"]
Oct 31 04:26:38.613153 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:26:38.613115    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35e6897c-0910-4afe-bdbb-7a19250f6735" path="/var/lib/kubelet/pods/35e6897c-0910-4afe-bdbb-7a19250f6735/volumes"
Oct 31 04:26:46.175763 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:26:46.175724    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:27:05.929124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:05.929086    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"]
Oct 31 04:27:05.929642 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:05.929298    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" podUID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" containerName="istio-proxy" containerID="cri-o://f0683081c625e0a191ccff9c6ca658d2e1883a12a6627ee09ba1d79794c7421f" gracePeriod=30
Oct 31 04:27:05.975241 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:05.975194    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"]
Oct 31 04:27:05.975529 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:05.975485    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx" podUID="013cbda1-7eb1-4fbe-8e03-c2de7b71928a" containerName="istio-proxy" containerID="cri-o://32b3117e60307f34566ea4fe20aea8a7f355014fa676c5f68f6e241ced695626" gracePeriod=30
Oct 31 04:27:06.289966 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:06.289865    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"]
Oct 31 04:27:06.302214 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:06.302178    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"]
Oct 31 04:27:06.311892 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:06.311841    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"]
Oct 31 04:27:11.228305 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.228274    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:27:11.271172 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.271143    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:27:11.273210 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273191    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-kube-api-access-hw4lc\" (UniqueName: \"kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-kube-api-access-hw4lc\") pod \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") "
Oct 31 04:27:11.273267 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273238    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-data\") pod \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") "
Oct 31 04:27:11.273267 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273263    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-certs\") pod \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") "
Oct 31 04:27:11.273350 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273289    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-credential-socket\") pod \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") "
Oct 31 04:27:11.273350 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273311    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-podinfo\") pod \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") "
Oct 31 04:27:11.273461 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273386    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-envoy\") pod \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") "
Oct 31 04:27:11.273614 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273550    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-token\" (UniqueName: \"kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-token\") pod \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") "
Oct 31 04:27:11.273614 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273528    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-certs" pod "99a09ce1-e619-406a-8097-ea1c5b8cbd3e" (UID: "99a09ce1-e619-406a-8097-ea1c5b8cbd3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.273614 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273582    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-data" pod "99a09ce1-e619-406a-8097-ea1c5b8cbd3e" (UID: "99a09ce1-e619-406a-8097-ea1c5b8cbd3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.273860 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273611    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-credential-socket" pod "99a09ce1-e619-406a-8097-ea1c5b8cbd3e" (UID: "99a09ce1-e619-406a-8097-ea1c5b8cbd3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.273860 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273615    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istiod-ca-cert\") pod \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") "
Oct 31 04:27:11.273860 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273681    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-socket\") pod \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\" (UID: \"99a09ce1-e619-406a-8097-ea1c5b8cbd3e\") "
Oct 31 04:27:11.274064 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273958    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-socket" pod "99a09ce1-e619-406a-8097-ea1c5b8cbd3e" (UID: "99a09ce1-e619-406a-8097-ea1c5b8cbd3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.274064 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.273977    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istiod-ca-cert" pod "99a09ce1-e619-406a-8097-ea1c5b8cbd3e" (UID: "99a09ce1-e619-406a-8097-ea1c5b8cbd3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:11.274178 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.274162    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.274236 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.274182    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.274236 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.274193    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.274236 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.274206    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.274236 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.274221    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.275631 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.275609    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-envoy" pod "99a09ce1-e619-406a-8097-ea1c5b8cbd3e" (UID: "99a09ce1-e619-406a-8097-ea1c5b8cbd3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.275733 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.275638    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-token" pod "99a09ce1-e619-406a-8097-ea1c5b8cbd3e" (UID: "99a09ce1-e619-406a-8097-ea1c5b8cbd3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:11.275923 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.275867    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-podinfo" pod "99a09ce1-e619-406a-8097-ea1c5b8cbd3e" (UID: "99a09ce1-e619-406a-8097-ea1c5b8cbd3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:27:11.276024 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.275901    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-kube-api-access-hw4lc" pod "99a09ce1-e619-406a-8097-ea1c5b8cbd3e" (UID: "99a09ce1-e619-406a-8097-ea1c5b8cbd3e"). InnerVolumeSpecName "kube-api-access-hw4lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:11.375401 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.375370    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-data\") pod \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") "
Oct 31 04:27:11.375593 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.375426    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-kube-api-access-s6fcf\" (UniqueName: \"kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-kube-api-access-s6fcf\") pod \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") "
Oct 31 04:27:11.375593 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.375453    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-podinfo\") pod \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") "
Oct 31 04:27:11.375593 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.375473    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istiod-ca-cert\") pod \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") "
Oct 31 04:27:11.375593 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.375504    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-token\" (UniqueName: \"kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-token\") pod \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") "
Oct 31 04:27:11.375593 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.375545    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-envoy\") pod \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") "
Oct 31 04:27:11.375593 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.375592    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-socket\") pod \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") "
Oct 31 04:27:11.375926 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.375660    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-certs\") pod \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") "
Oct 31 04:27:11.375926 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.375759    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-credential-socket\") pod \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\" (UID: \"013cbda1-7eb1-4fbe-8e03-c2de7b71928a\") "
Oct 31 04:27:11.375926 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.375823    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-data" pod "013cbda1-7eb1-4fbe-8e03-c2de7b71928a" (UID: "013cbda1-7eb1-4fbe-8e03-c2de7b71928a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.376095 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.375993    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istiod-ca-cert" pod "013cbda1-7eb1-4fbe-8e03-c2de7b71928a" (UID: "013cbda1-7eb1-4fbe-8e03-c2de7b71928a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:11.376339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.376206    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-credential-socket" pod "013cbda1-7eb1-4fbe-8e03-c2de7b71928a" (UID: "013cbda1-7eb1-4fbe-8e03-c2de7b71928a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.376339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.376270    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-socket" pod "013cbda1-7eb1-4fbe-8e03-c2de7b71928a" (UID: "013cbda1-7eb1-4fbe-8e03-c2de7b71928a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.376339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.376339    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-certs" pod "013cbda1-7eb1-4fbe-8e03-c2de7b71928a" (UID: "013cbda1-7eb1-4fbe-8e03-c2de7b71928a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.376607 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.376360    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.376607 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.376380    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.376607 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.376395    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.376607 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.376411    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw4lc\" (UniqueName: \"kubernetes.io/projected/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-kube-api-access-hw4lc\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.376607 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.376427    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.376607 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.376442    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.376607 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.376456    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a09ce1-e619-406a-8097-ea1c5b8cbd3e-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.377893 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.377852    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-podinfo" pod "013cbda1-7eb1-4fbe-8e03-c2de7b71928a" (UID: "013cbda1-7eb1-4fbe-8e03-c2de7b71928a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:27:11.378017 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.377869    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-token" pod "013cbda1-7eb1-4fbe-8e03-c2de7b71928a" (UID: "013cbda1-7eb1-4fbe-8e03-c2de7b71928a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:11.378017 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.377923    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-kube-api-access-s6fcf" pod "013cbda1-7eb1-4fbe-8e03-c2de7b71928a" (UID: "013cbda1-7eb1-4fbe-8e03-c2de7b71928a"). InnerVolumeSpecName "kube-api-access-s6fcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:11.378103 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.378076    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-envoy" pod "013cbda1-7eb1-4fbe-8e03-c2de7b71928a" (UID: "013cbda1-7eb1-4fbe-8e03-c2de7b71928a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.442649 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.442612    5415 generic.go:356] "Generic (PLEG): container finished" podID="013cbda1-7eb1-4fbe-8e03-c2de7b71928a" containerID="32b3117e60307f34566ea4fe20aea8a7f355014fa676c5f68f6e241ced695626" exitCode=0
Oct 31 04:27:11.442863 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.442668    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"
Oct 31 04:27:11.442863 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.442669    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx" event={"ID":"013cbda1-7eb1-4fbe-8e03-c2de7b71928a","Type":"ContainerDied","Data":"32b3117e60307f34566ea4fe20aea8a7f355014fa676c5f68f6e241ced695626"}
Oct 31 04:27:11.442863 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.442800    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx" event={"ID":"013cbda1-7eb1-4fbe-8e03-c2de7b71928a","Type":"ContainerDied","Data":"e6db6a550c9e7c20d9e4e91a35fbd6e01a679757f4204b4ad9db15d28431cc57"}
Oct 31 04:27:11.442863 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.442827    5415 scope.go:122] "RemoveContainer" containerID="32b3117e60307f34566ea4fe20aea8a7f355014fa676c5f68f6e241ced695626"
Oct 31 04:27:11.444577 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.444551    5415 generic.go:356] "Generic (PLEG): container finished" podID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" containerID="f0683081c625e0a191ccff9c6ca658d2e1883a12a6627ee09ba1d79794c7421f" exitCode=0
Oct 31 04:27:11.444734 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.444623    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"
Oct 31 04:27:11.444734 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.444657    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" event={"ID":"99a09ce1-e619-406a-8097-ea1c5b8cbd3e","Type":"ContainerDied","Data":"f0683081c625e0a191ccff9c6ca658d2e1883a12a6627ee09ba1d79794c7421f"}
Oct 31 04:27:11.444734 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.444704    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj" event={"ID":"99a09ce1-e619-406a-8097-ea1c5b8cbd3e","Type":"ContainerDied","Data":"db7172fe7917659ffec685f53b76bd3e01da33c2d2ce7fb36bf72256e8dbee45"}
Oct 31 04:27:11.455151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.455043    5415 scope.go:122] "RemoveContainer" containerID="32b3117e60307f34566ea4fe20aea8a7f355014fa676c5f68f6e241ced695626"
Oct 31 04:27:11.455478 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:27:11.455451    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b3117e60307f34566ea4fe20aea8a7f355014fa676c5f68f6e241ced695626\": container with ID starting with 32b3117e60307f34566ea4fe20aea8a7f355014fa676c5f68f6e241ced695626 not found: ID does not exist" containerID="32b3117e60307f34566ea4fe20aea8a7f355014fa676c5f68f6e241ced695626"
Oct 31 04:27:11.455569 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.455491    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b3117e60307f34566ea4fe20aea8a7f355014fa676c5f68f6e241ced695626"} err="failed to get container status \"32b3117e60307f34566ea4fe20aea8a7f355014fa676c5f68f6e241ced695626\": rpc error: code = NotFound desc = could not find container \"32b3117e60307f34566ea4fe20aea8a7f355014fa676c5f68f6e241ced695626\": container with ID starting with 32b3117e60307f34566ea4fe20aea8a7f355014fa676c5f68f6e241ced695626 not found: ID does not exist"
Oct 31 04:27:11.455569 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.455517    5415 scope.go:122] "RemoveContainer" containerID="f0683081c625e0a191ccff9c6ca658d2e1883a12a6627ee09ba1d79794c7421f"
Oct 31 04:27:11.464938 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.464918    5415 scope.go:122] "RemoveContainer" containerID="f0683081c625e0a191ccff9c6ca658d2e1883a12a6627ee09ba1d79794c7421f"
Oct 31 04:27:11.465260 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:27:11.465241    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0683081c625e0a191ccff9c6ca658d2e1883a12a6627ee09ba1d79794c7421f\": container with ID starting with f0683081c625e0a191ccff9c6ca658d2e1883a12a6627ee09ba1d79794c7421f not found: ID does not exist" containerID="f0683081c625e0a191ccff9c6ca658d2e1883a12a6627ee09ba1d79794c7421f"
Oct 31 04:27:11.465340 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.465272    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0683081c625e0a191ccff9c6ca658d2e1883a12a6627ee09ba1d79794c7421f"} err="failed to get container status \"f0683081c625e0a191ccff9c6ca658d2e1883a12a6627ee09ba1d79794c7421f\": rpc error: code = NotFound desc = could not find container \"f0683081c625e0a191ccff9c6ca658d2e1883a12a6627ee09ba1d79794c7421f\": container with ID starting with f0683081c625e0a191ccff9c6ca658d2e1883a12a6627ee09ba1d79794c7421f not found: ID does not exist"
Oct 31 04:27:11.477929 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.477869    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6fcf\" (UniqueName: \"kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-kube-api-access-s6fcf\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.477929 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.477930    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.478110 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.477943    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.478110 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.477952    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.478110 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.477961    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.478110 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.477970    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013cbda1-7eb1-4fbe-8e03-c2de7b71928a-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.606783 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.606748    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"]
Oct 31 04:27:11.612986 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.612952    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bgjqj"]
Oct 31 04:27:11.717659 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.717582    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"]
Oct 31 04:27:11.724188 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:11.724152    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-npjlx"]
Oct 31 04:27:12.612762 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:12.607186    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013cbda1-7eb1-4fbe-8e03-c2de7b71928a" path="/var/lib/kubelet/pods/013cbda1-7eb1-4fbe-8e03-c2de7b71928a/volumes"
Oct 31 04:27:12.612762 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:12.607944    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" path="/var/lib/kubelet/pods/99a09ce1-e619-406a-8097-ea1c5b8cbd3e/volumes"
Oct 31 04:27:16.175542 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:27:16.175460    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:27:21.311794 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.311747    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm" podUID="f39ffc5b-0fbf-41ff-91c9-9242538ed564" containerName="main" containerID="cri-o://f76f08e1fac660c6c5dbe8288e0be6adc22d071a5be4e0ced54bdcce506528c8" gracePeriod=45
Oct 31 04:27:21.323633 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.323574    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerName="main" containerID="cri-o://de39d696103a67b4320410a5ab7e13cb91f240cf1bc781ccdb98c0087cbbd1f9" gracePeriod=45
Oct 31 04:27:21.333320 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.333288    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt" podUID="7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" containerName="main" containerID="cri-o://6a8f95ba370bde5187643f288281db6ca064677126d024b8de9a453d9c2f7827" gracePeriod=45
Oct 31 04:27:21.505017 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.503741    5415 generic.go:356] "Generic (PLEG): container finished" podID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerID="de39d696103a67b4320410a5ab7e13cb91f240cf1bc781ccdb98c0087cbbd1f9" exitCode=0
Oct 31 04:27:21.505017 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.503822    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" event={"ID":"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150","Type":"ContainerDied","Data":"de39d696103a67b4320410a5ab7e13cb91f240cf1bc781ccdb98c0087cbbd1f9"}
Oct 31 04:27:21.508915 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.505354    5415 generic.go:356] "Generic (PLEG): container finished" podID="f39ffc5b-0fbf-41ff-91c9-9242538ed564" containerID="f76f08e1fac660c6c5dbe8288e0be6adc22d071a5be4e0ced54bdcce506528c8" exitCode=0
Oct 31 04:27:21.508915 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.505494    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm" event={"ID":"f39ffc5b-0fbf-41ff-91c9-9242538ed564","Type":"ContainerDied","Data":"f76f08e1fac660c6c5dbe8288e0be6adc22d071a5be4e0ced54bdcce506528c8"}
Oct 31 04:27:21.508915 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.506892    5415 generic.go:356] "Generic (PLEG): container finished" podID="7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" containerID="6a8f95ba370bde5187643f288281db6ca064677126d024b8de9a453d9c2f7827" exitCode=0
Oct 31 04:27:21.508915 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.507189    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt" event={"ID":"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a","Type":"ContainerDied","Data":"6a8f95ba370bde5187643f288281db6ca064677126d024b8de9a453d9c2f7827"}
Oct 31 04:27:21.517317 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.516814    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm" podUID="f39ffc5b-0fbf-41ff-91c9-9242538ed564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 31 04:27:21.608516 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.608485    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 04:27:21.641074 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.641042    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 04:27:21.644308 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.644286    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 04:27:21.683796 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.683762    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tmp-dir\") pod \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") "
Oct 31 04:27:21.684020 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.683937    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tls-certs\") pod \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") "
Oct 31 04:27:21.684020 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.683969    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-model-cache\") pod \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") "
Oct 31 04:27:21.684020 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.683991    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-home\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-home\") pod \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") "
Oct 31 04:27:21.684203 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.684062    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-dshm\") pod \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") "
Oct 31 04:27:21.684203 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.684094    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-kube-api-access-v72w8\" (UniqueName: \"kubernetes.io/projected/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-kube-api-access-v72w8\") pod \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\" (UID: \"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150\") "
Oct 31 04:27:21.684203 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.684105    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tmp-dir" pod "2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" (UID: "2d1ad969-4b4b-4178-a5a9-7f76d8cd2150"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.684376 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.684251    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-model-cache" pod "2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" (UID: "2d1ad969-4b4b-4178-a5a9-7f76d8cd2150"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.684574 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.684551    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.684712 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.684648    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.684712 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.684648    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-home" pod "2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" (UID: "2d1ad969-4b4b-4178-a5a9-7f76d8cd2150"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.686357 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.686337    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-dshm" pod "2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" (UID: "2d1ad969-4b4b-4178-a5a9-7f76d8cd2150"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.686498 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.686480    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-kube-api-access-v72w8" pod "2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" (UID: "2d1ad969-4b4b-4178-a5a9-7f76d8cd2150"). InnerVolumeSpecName "kube-api-access-v72w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:21.686617 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.686605    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tls-certs" pod "2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" (UID: "2d1ad969-4b4b-4178-a5a9-7f76d8cd2150"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:21.785682 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.785496    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f39ffc5b-0fbf-41ff-91c9-9242538ed564-kube-api-access-8z6pk\" (UniqueName: \"kubernetes.io/projected/f39ffc5b-0fbf-41ff-91c9-9242538ed564-kube-api-access-8z6pk\") pod \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") "
Oct 31 04:27:21.785682 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.785627    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-home\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-home\") pod \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") "
Oct 31 04:27:21.785682 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.785659    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tls-certs\") pod \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") "
Oct 31 04:27:21.785682 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.785687    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-kube-api-access-mn798\" (UniqueName: \"kubernetes.io/projected/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-kube-api-access-mn798\") pod \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") "
Oct 31 04:27:21.786470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.785742    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tls-certs\" (UniqueName: \"kubernetes.io/secret/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tls-certs\") pod \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") "
Oct 31 04:27:21.786470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.785780    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-model-cache\") pod \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") "
Oct 31 04:27:21.786470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.785822    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-model-cache\") pod \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") "
Oct 31 04:27:21.786470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.785849    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tmp-dir\") pod \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") "
Oct 31 04:27:21.786470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.785916    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tmp-dir\") pod \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") "
Oct 31 04:27:21.786470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.785945    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-dshm\") pod \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\" (UID: \"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a\") "
Oct 31 04:27:21.786470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.785991    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-home\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-home\") pod \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") "
Oct 31 04:27:21.786470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786036    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-dshm\") pod \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\" (UID: \"f39ffc5b-0fbf-41ff-91c9-9242538ed564\") "
Oct 31 04:27:21.786470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786092    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-home" pod "7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" (UID: "7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.786470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786113    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-model-cache" pod "7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" (UID: "7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.786470 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786316    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tmp-dir" pod "7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" (UID: "7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.787124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786510    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-home" pod "f39ffc5b-0fbf-41ff-91c9-9242538ed564" (UID: "f39ffc5b-0fbf-41ff-91c9-9242538ed564"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.787124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786513    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.787124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786552    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.787124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786567    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.787124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786582    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.787124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786667    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-model-cache" pod "f39ffc5b-0fbf-41ff-91c9-9242538ed564" (UID: "f39ffc5b-0fbf-41ff-91c9-9242538ed564"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.787124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786678    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tmp-dir" pod "f39ffc5b-0fbf-41ff-91c9-9242538ed564" (UID: "f39ffc5b-0fbf-41ff-91c9-9242538ed564"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.787124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786736    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v72w8\" (UniqueName: \"kubernetes.io/projected/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150-kube-api-access-v72w8\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.787124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786759    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.787124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.786774    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.788466 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.788421    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-dshm" pod "f39ffc5b-0fbf-41ff-91c9-9242538ed564" (UID: "f39ffc5b-0fbf-41ff-91c9-9242538ed564"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.788623 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.788603    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tls-certs" pod "7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" (UID: "7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:21.788728 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.788665    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39ffc5b-0fbf-41ff-91c9-9242538ed564-kube-api-access-8z6pk" pod "f39ffc5b-0fbf-41ff-91c9-9242538ed564" (UID: "f39ffc5b-0fbf-41ff-91c9-9242538ed564"). InnerVolumeSpecName "kube-api-access-8z6pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:21.788853 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.788833    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-kube-api-access-mn798" pod "7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" (UID: "7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a"). InnerVolumeSpecName "kube-api-access-mn798". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:21.789073 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.789042    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tls-certs" pod "f39ffc5b-0fbf-41ff-91c9-9242538ed564" (UID: "f39ffc5b-0fbf-41ff-91c9-9242538ed564"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:21.789659 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.789627    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-dshm" pod "7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" (UID: "7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.888222 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.888134    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.888222 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.888169    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z6pk\" (UniqueName: \"kubernetes.io/projected/f39ffc5b-0fbf-41ff-91c9-9242538ed564-kube-api-access-8z6pk\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.888222 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.888202    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.888222 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.888217    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn798\" (UniqueName: \"kubernetes.io/projected/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-kube-api-access-mn798\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.888564 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.888232    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.888564 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.888245    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.888564 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.888259    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.888564 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.888268    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.888564 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:21.888276    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f39ffc5b-0fbf-41ff-91c9-9242538ed564-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:22.513526 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.513499    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"
Oct 31 04:27:22.514043 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.513537    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt" event={"ID":"7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a","Type":"ContainerDied","Data":"eb4fff0cfb75d411362b4e2a2edbbe3942b4c6778f53702d5cfaca9de79dba9d"}
Oct 31 04:27:22.514043 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.513597    5415 scope.go:122] "RemoveContainer" containerID="6a8f95ba370bde5187643f288281db6ca064677126d024b8de9a453d9c2f7827"
Oct 31 04:27:22.515906 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.515316    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"
Oct 31 04:27:22.515906 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.515324    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps" event={"ID":"2d1ad969-4b4b-4178-a5a9-7f76d8cd2150","Type":"ContainerDied","Data":"aa270eec466e8d5c3c1e1874d67642538f8fd16cfa45630656f79c0790f38ec5"}
Oct 31 04:27:22.518926 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.517008    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm" event={"ID":"f39ffc5b-0fbf-41ff-91c9-9242538ed564","Type":"ContainerDied","Data":"1e736083b4c58b5b805e04f84e608764863496cc229e6b6b6f3d0c538e54209d"}
Oct 31 04:27:22.518926 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.517016    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"
Oct 31 04:27:22.525399 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.525364    5415 scope.go:122] "RemoveContainer" containerID="de39d696103a67b4320410a5ab7e13cb91f240cf1bc781ccdb98c0087cbbd1f9"
Oct 31 04:27:22.534847 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.534825    5415 scope.go:122] "RemoveContainer" containerID="f76f08e1fac660c6c5dbe8288e0be6adc22d071a5be4e0ced54bdcce506528c8"
Oct 31 04:27:22.666829 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.666797    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"]
Oct 31 04:27:22.714292 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.714254    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/simulator-w1-d39257-kserve-559566dd5d-74jrt"]
Oct 31 04:27:22.816670 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.816620    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"]
Oct 31 04:27:22.822049 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.822018    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/premium-w1-d39257-kserve-754b5bb4c-f2sps"]
Oct 31 04:27:22.923993 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.923908    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"]
Oct 31 04:27:22.929814 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:22.929780    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/distinct-2-w1-d39257-kserve-648986f497-zrkdm"]
Oct 31 04:27:24.605081 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:24.605048    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" path="/var/lib/kubelet/pods/2d1ad969-4b4b-4178-a5a9-7f76d8cd2150/volumes"
Oct 31 04:27:24.605505 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:24.605491    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" path="/var/lib/kubelet/pods/7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a/volumes"
Oct 31 04:27:24.605932 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:24.605913    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f39ffc5b-0fbf-41ff-91c9-9242538ed564" path="/var/lib/kubelet/pods/f39ffc5b-0fbf-41ff-91c9-9242538ed564/volumes"
Oct 31 04:27:35.489967 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.489918    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"]
Oct 31 04:27:35.490489 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.490207    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g" podUID="2ad27d6d-957f-4a3b-bcef-5a1bab901463" containerName="maas-api" containerID="cri-o://8396114e460bb74c400becf1eeb25839051fb5db24c19d7a62f0b5d5b20623a3" gracePeriod=30
Oct 31 04:27:35.764204 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.764165    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 04:27:35.817278 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.817232    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-metrics-certs\") pod \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") "
Oct 31 04:27:35.817278 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.817287    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-maas-api-tls\") pod \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") "
Oct 31 04:27:35.817584 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.817401    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad27d6d-957f-4a3b-bcef-5a1bab901463-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad27d6d-957f-4a3b-bcef-5a1bab901463-openshift-service-ca\") pod \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") "
Oct 31 04:27:35.817741 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.817715    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad27d6d-957f-4a3b-bcef-5a1bab901463-openshift-service-ca" pod "2ad27d6d-957f-4a3b-bcef-5a1bab901463" (UID: "2ad27d6d-957f-4a3b-bcef-5a1bab901463"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:35.817902 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.817798    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad27d6d-957f-4a3b-bcef-5a1bab901463-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:35.819484 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.819465    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-maas-api-tls" pod "2ad27d6d-957f-4a3b-bcef-5a1bab901463" (UID: "2ad27d6d-957f-4a3b-bcef-5a1bab901463"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:35.819781 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.819766    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-metrics-certs" pod "2ad27d6d-957f-4a3b-bcef-5a1bab901463" (UID: "2ad27d6d-957f-4a3b-bcef-5a1bab901463"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:35.918585 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.918546    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad27d6d-957f-4a3b-bcef-5a1bab901463-kube-api-access-7245g\" (UniqueName: \"kubernetes.io/projected/2ad27d6d-957f-4a3b-bcef-5a1bab901463-kube-api-access-7245g\") pod \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\" (UID: \"2ad27d6d-957f-4a3b-bcef-5a1bab901463\") "
Oct 31 04:27:35.918927 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.918909    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:35.919010 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.918931    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ad27d6d-957f-4a3b-bcef-5a1bab901463-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:35.920780 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:35.920750    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad27d6d-957f-4a3b-bcef-5a1bab901463-kube-api-access-7245g" pod "2ad27d6d-957f-4a3b-bcef-5a1bab901463" (UID: "2ad27d6d-957f-4a3b-bcef-5a1bab901463"). InnerVolumeSpecName "kube-api-access-7245g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:36.020135 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:36.020053    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7245g\" (UniqueName: \"kubernetes.io/projected/2ad27d6d-957f-4a3b-bcef-5a1bab901463-kube-api-access-7245g\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:36.596580 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:36.596547    5415 generic.go:356] "Generic (PLEG): container finished" podID="2ad27d6d-957f-4a3b-bcef-5a1bab901463" containerID="8396114e460bb74c400becf1eeb25839051fb5db24c19d7a62f0b5d5b20623a3" exitCode=0
Oct 31 04:27:36.597024 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:36.596595    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"
Oct 31 04:27:36.597024 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:36.596605    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g" event={"ID":"2ad27d6d-957f-4a3b-bcef-5a1bab901463","Type":"ContainerDied","Data":"8396114e460bb74c400becf1eeb25839051fb5db24c19d7a62f0b5d5b20623a3"}
Oct 31 04:27:36.597024 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:36.596649    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g" event={"ID":"2ad27d6d-957f-4a3b-bcef-5a1bab901463","Type":"ContainerDied","Data":"fc82b6e38cc6f003019c8625877dc0137a30cc0b831989cea53d1b8ac88a1680"}
Oct 31 04:27:36.597024 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:36.596670    5415 scope.go:122] "RemoveContainer" containerID="8396114e460bb74c400becf1eeb25839051fb5db24c19d7a62f0b5d5b20623a3"
Oct 31 04:27:36.607301 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:36.607281    5415 scope.go:122] "RemoveContainer" containerID="8396114e460bb74c400becf1eeb25839051fb5db24c19d7a62f0b5d5b20623a3"
Oct 31 04:27:36.607615 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:27:36.607594    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8396114e460bb74c400becf1eeb25839051fb5db24c19d7a62f0b5d5b20623a3\": container with ID starting with 8396114e460bb74c400becf1eeb25839051fb5db24c19d7a62f0b5d5b20623a3 not found: ID does not exist" containerID="8396114e460bb74c400becf1eeb25839051fb5db24c19d7a62f0b5d5b20623a3"
Oct 31 04:27:36.607710 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:36.607625    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8396114e460bb74c400becf1eeb25839051fb5db24c19d7a62f0b5d5b20623a3"} err="failed to get container status \"8396114e460bb74c400becf1eeb25839051fb5db24c19d7a62f0b5d5b20623a3\": rpc error: code = NotFound desc = could not find container \"8396114e460bb74c400becf1eeb25839051fb5db24c19d7a62f0b5d5b20623a3\": container with ID starting with 8396114e460bb74c400becf1eeb25839051fb5db24c19d7a62f0b5d5b20623a3 not found: ID does not exist"
Oct 31 04:27:36.750430 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:36.750384    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"]
Oct 31 04:27:36.753804 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:36.753774    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2339ed-5fc644fd46-pwn8g"]
Oct 31 04:27:38.605406 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:38.605368    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad27d6d-957f-4a3b-bcef-5a1bab901463" path="/var/lib/kubelet/pods/2ad27d6d-957f-4a3b-bcef-5a1bab901463/volumes"
Oct 31 04:27:46.176867 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:27:46.176825    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:27:58.655207 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:58.655164    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"]
Oct 31 04:27:58.655721 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:58.655439    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw" podUID="b54532f2-51ef-4ade-a041-b3c0c74105bf" containerName="maas-api" containerID="cri-o://ddd220bfdee4b0ca398de07f0277c44df54e72b44f6050c5c68ffad64b5b0d60" gracePeriod=30
Oct 31 04:27:58.921662 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:58.921587    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 04:27:59.066518 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.066471    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-maas-api-tls\") pod \"b54532f2-51ef-4ade-a041-b3c0c74105bf\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") "
Oct 31 04:27:59.066726 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.066620    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-metrics-certs\") pod \"b54532f2-51ef-4ade-a041-b3c0c74105bf\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") "
Oct 31 04:27:59.066726 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.066670    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b54532f2-51ef-4ade-a041-b3c0c74105bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54532f2-51ef-4ade-a041-b3c0c74105bf-openshift-service-ca\") pod \"b54532f2-51ef-4ade-a041-b3c0c74105bf\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") "
Oct 31 04:27:59.066868 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.066749    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b54532f2-51ef-4ade-a041-b3c0c74105bf-kube-api-access-c85hm\" (UniqueName: \"kubernetes.io/projected/b54532f2-51ef-4ade-a041-b3c0c74105bf-kube-api-access-c85hm\") pod \"b54532f2-51ef-4ade-a041-b3c0c74105bf\" (UID: \"b54532f2-51ef-4ade-a041-b3c0c74105bf\") "
Oct 31 04:27:59.067133 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.067107    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54532f2-51ef-4ade-a041-b3c0c74105bf-openshift-service-ca" pod "b54532f2-51ef-4ade-a041-b3c0c74105bf" (UID: "b54532f2-51ef-4ade-a041-b3c0c74105bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:59.068797 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.068769    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-metrics-certs" pod "b54532f2-51ef-4ade-a041-b3c0c74105bf" (UID: "b54532f2-51ef-4ade-a041-b3c0c74105bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:59.068952 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.068792    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-maas-api-tls" pod "b54532f2-51ef-4ade-a041-b3c0c74105bf" (UID: "b54532f2-51ef-4ade-a041-b3c0c74105bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:59.068952 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.068908    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54532f2-51ef-4ade-a041-b3c0c74105bf-kube-api-access-c85hm" pod "b54532f2-51ef-4ade-a041-b3c0c74105bf" (UID: "b54532f2-51ef-4ade-a041-b3c0c74105bf"). InnerVolumeSpecName "kube-api-access-c85hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:59.168634 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.168596    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c85hm\" (UniqueName: \"kubernetes.io/projected/b54532f2-51ef-4ade-a041-b3c0c74105bf-kube-api-access-c85hm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:59.168634 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.168631    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:59.168634 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.168642    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b54532f2-51ef-4ade-a041-b3c0c74105bf-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:59.168634 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.168652    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54532f2-51ef-4ade-a041-b3c0c74105bf-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:59.725810 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.725775    5415 generic.go:356] "Generic (PLEG): container finished" podID="b54532f2-51ef-4ade-a041-b3c0c74105bf" containerID="ddd220bfdee4b0ca398de07f0277c44df54e72b44f6050c5c68ffad64b5b0d60" exitCode=0
Oct 31 04:27:59.726356 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.725827    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"
Oct 31 04:27:59.726356 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.725859    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw" event={"ID":"b54532f2-51ef-4ade-a041-b3c0c74105bf","Type":"ContainerDied","Data":"ddd220bfdee4b0ca398de07f0277c44df54e72b44f6050c5c68ffad64b5b0d60"}
Oct 31 04:27:59.726356 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.725931    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw" event={"ID":"b54532f2-51ef-4ade-a041-b3c0c74105bf","Type":"ContainerDied","Data":"745e04bf7c46aa847f1bd981d080da7fc7dea5af2ed4598e879f2a9569cfb265"}
Oct 31 04:27:59.726356 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.725955    5415 scope.go:122] "RemoveContainer" containerID="ddd220bfdee4b0ca398de07f0277c44df54e72b44f6050c5c68ffad64b5b0d60"
Oct 31 04:27:59.735795 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.735775    5415 scope.go:122] "RemoveContainer" containerID="ddd220bfdee4b0ca398de07f0277c44df54e72b44f6050c5c68ffad64b5b0d60"
Oct 31 04:27:59.736139 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:27:59.736118    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd220bfdee4b0ca398de07f0277c44df54e72b44f6050c5c68ffad64b5b0d60\": container with ID starting with ddd220bfdee4b0ca398de07f0277c44df54e72b44f6050c5c68ffad64b5b0d60 not found: ID does not exist" containerID="ddd220bfdee4b0ca398de07f0277c44df54e72b44f6050c5c68ffad64b5b0d60"
Oct 31 04:27:59.736221 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.736154    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd220bfdee4b0ca398de07f0277c44df54e72b44f6050c5c68ffad64b5b0d60"} err="failed to get container status \"ddd220bfdee4b0ca398de07f0277c44df54e72b44f6050c5c68ffad64b5b0d60\": rpc error: code = NotFound desc = could not find container \"ddd220bfdee4b0ca398de07f0277c44df54e72b44f6050c5c68ffad64b5b0d60\": container with ID starting with ddd220bfdee4b0ca398de07f0277c44df54e72b44f6050c5c68ffad64b5b0d60 not found: ID does not exist"
Oct 31 04:27:59.882361 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.882325    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"]
Oct 31 04:27:59.923447 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:27:59.923410    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw"]
Oct 31 04:28:00.604974 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:00.604928    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b54532f2-51ef-4ade-a041-b3c0c74105bf" path="/var/lib/kubelet/pods/b54532f2-51ef-4ade-a041-b3c0c74105bf/volumes"
Oct 31 04:28:01.916271 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:01.916216    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d39257-7766d88b76-p5lmw" podUID="b54532f2-51ef-4ade-a041-b3c0c74105bf" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.44:8443/health\": context deadline exceeded"
Oct 31 04:28:05.423092 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.423055    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"]
Oct 31 04:28:05.423567 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.423337    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2" podUID="abc2f709-4ac0-40c9-bea6-5306df98321c" containerName="maas-api" containerID="cri-o://f8cbe6888d8f3f675a6b2fd264b289bfffe1370ca3d902d27a08404c0134b2a0" gracePeriod=30
Oct 31 04:28:05.703319 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.703283    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 04:28:05.737298 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.737262    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc2f709-4ac0-40c9-bea6-5306df98321c-kube-api-access-mvlck\" (UniqueName: \"kubernetes.io/projected/abc2f709-4ac0-40c9-bea6-5306df98321c-kube-api-access-mvlck\") pod \"abc2f709-4ac0-40c9-bea6-5306df98321c\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") "
Oct 31 04:28:05.737496 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.737325    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc2f709-4ac0-40c9-bea6-5306df98321c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc2f709-4ac0-40c9-bea6-5306df98321c-openshift-service-ca\") pod \"abc2f709-4ac0-40c9-bea6-5306df98321c\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") "
Oct 31 04:28:05.737496 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.737372    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-metrics-certs\") pod \"abc2f709-4ac0-40c9-bea6-5306df98321c\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") "
Oct 31 04:28:05.737496 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.737460    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-maas-api-tls\") pod \"abc2f709-4ac0-40c9-bea6-5306df98321c\" (UID: \"abc2f709-4ac0-40c9-bea6-5306df98321c\") "
Oct 31 04:28:05.737802 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.737770    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc2f709-4ac0-40c9-bea6-5306df98321c-openshift-service-ca" pod "abc2f709-4ac0-40c9-bea6-5306df98321c" (UID: "abc2f709-4ac0-40c9-bea6-5306df98321c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:05.739597 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.739563    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-metrics-certs" pod "abc2f709-4ac0-40c9-bea6-5306df98321c" (UID: "abc2f709-4ac0-40c9-bea6-5306df98321c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:05.739597 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.739593    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc2f709-4ac0-40c9-bea6-5306df98321c-kube-api-access-mvlck" pod "abc2f709-4ac0-40c9-bea6-5306df98321c" (UID: "abc2f709-4ac0-40c9-bea6-5306df98321c"). InnerVolumeSpecName "kube-api-access-mvlck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:05.739736 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.739659    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-maas-api-tls" pod "abc2f709-4ac0-40c9-bea6-5306df98321c" (UID: "abc2f709-4ac0-40c9-bea6-5306df98321c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:05.758940 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.758913    5415 generic.go:356] "Generic (PLEG): container finished" podID="abc2f709-4ac0-40c9-bea6-5306df98321c" containerID="f8cbe6888d8f3f675a6b2fd264b289bfffe1370ca3d902d27a08404c0134b2a0" exitCode=0
Oct 31 04:28:05.759091 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.758955    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"
Oct 31 04:28:05.759091 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.758993    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2" event={"ID":"abc2f709-4ac0-40c9-bea6-5306df98321c","Type":"ContainerDied","Data":"f8cbe6888d8f3f675a6b2fd264b289bfffe1370ca3d902d27a08404c0134b2a0"}
Oct 31 04:28:05.759091 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.759055    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2" event={"ID":"abc2f709-4ac0-40c9-bea6-5306df98321c","Type":"ContainerDied","Data":"e917c77be1c3ce519f77c10056b01ac2f03b0f3a22715e996ce9fd195ceb387a"}
Oct 31 04:28:05.759091 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.759085    5415 scope.go:122] "RemoveContainer" containerID="f8cbe6888d8f3f675a6b2fd264b289bfffe1370ca3d902d27a08404c0134b2a0"
Oct 31 04:28:05.769112 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.769091    5415 scope.go:122] "RemoveContainer" containerID="f8cbe6888d8f3f675a6b2fd264b289bfffe1370ca3d902d27a08404c0134b2a0"
Oct 31 04:28:05.769406 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:28:05.769386    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8cbe6888d8f3f675a6b2fd264b289bfffe1370ca3d902d27a08404c0134b2a0\": container with ID starting with f8cbe6888d8f3f675a6b2fd264b289bfffe1370ca3d902d27a08404c0134b2a0 not found: ID does not exist" containerID="f8cbe6888d8f3f675a6b2fd264b289bfffe1370ca3d902d27a08404c0134b2a0"
Oct 31 04:28:05.769459 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.769417    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8cbe6888d8f3f675a6b2fd264b289bfffe1370ca3d902d27a08404c0134b2a0"} err="failed to get container status \"f8cbe6888d8f3f675a6b2fd264b289bfffe1370ca3d902d27a08404c0134b2a0\": rpc error: code = NotFound desc = could not find container \"f8cbe6888d8f3f675a6b2fd264b289bfffe1370ca3d902d27a08404c0134b2a0\": container with ID starting with f8cbe6888d8f3f675a6b2fd264b289bfffe1370ca3d902d27a08404c0134b2a0 not found: ID does not exist"
Oct 31 04:28:05.838432 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.838394    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:05.838432 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.838435    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvlck\" (UniqueName: \"kubernetes.io/projected/abc2f709-4ac0-40c9-bea6-5306df98321c-kube-api-access-mvlck\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:05.838636 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.838452    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc2f709-4ac0-40c9-bea6-5306df98321c-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:05.838636 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.838467    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc2f709-4ac0-40c9-bea6-5306df98321c-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:05.919718 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.919677    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"]
Oct 31 04:28:05.923521 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:05.923486    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-502ec2-67c6db9c48-zjqp2"]
Oct 31 04:28:06.606014 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:06.605976    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc2f709-4ac0-40c9-bea6-5306df98321c" path="/var/lib/kubelet/pods/abc2f709-4ac0-40c9-bea6-5306df98321c/volumes"
Oct 31 04:28:11.401578 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.401535    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"]
Oct 31 04:28:11.402044 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.401741    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d" podUID="e886c2bd-b363-480c-95c4-b889527cfa22" containerName="maas-api" containerID="cri-o://2e194cd144ddb7a31575cd5e434233efa0b2e33cbd13aa6af49fe9a2d3d9b486" gracePeriod=30
Oct 31 04:28:11.665315 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.665235    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 04:28:11.692853 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.692806    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e886c2bd-b363-480c-95c4-b889527cfa22-kube-api-access-2z9cc\" (UniqueName: \"kubernetes.io/projected/e886c2bd-b363-480c-95c4-b889527cfa22-kube-api-access-2z9cc\") pod \"e886c2bd-b363-480c-95c4-b889527cfa22\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") "
Oct 31 04:28:11.692853 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.692855    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-maas-api-tls\") pod \"e886c2bd-b363-480c-95c4-b889527cfa22\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") "
Oct 31 04:28:11.693142 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.692963    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-metrics-certs\") pod \"e886c2bd-b363-480c-95c4-b889527cfa22\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") "
Oct 31 04:28:11.693142 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.693012    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e886c2bd-b363-480c-95c4-b889527cfa22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e886c2bd-b363-480c-95c4-b889527cfa22-openshift-service-ca\") pod \"e886c2bd-b363-480c-95c4-b889527cfa22\" (UID: \"e886c2bd-b363-480c-95c4-b889527cfa22\") "
Oct 31 04:28:11.693476 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.693445    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e886c2bd-b363-480c-95c4-b889527cfa22-openshift-service-ca" pod "e886c2bd-b363-480c-95c4-b889527cfa22" (UID: "e886c2bd-b363-480c-95c4-b889527cfa22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:11.695297 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.695268    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-maas-api-tls" pod "e886c2bd-b363-480c-95c4-b889527cfa22" (UID: "e886c2bd-b363-480c-95c4-b889527cfa22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:11.695630 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.695606    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-metrics-certs" pod "e886c2bd-b363-480c-95c4-b889527cfa22" (UID: "e886c2bd-b363-480c-95c4-b889527cfa22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:11.695724 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.695643    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e886c2bd-b363-480c-95c4-b889527cfa22-kube-api-access-2z9cc" pod "e886c2bd-b363-480c-95c4-b889527cfa22" (UID: "e886c2bd-b363-480c-95c4-b889527cfa22"). InnerVolumeSpecName "kube-api-access-2z9cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:11.794339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.794120    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:11.794339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.794156    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e886c2bd-b363-480c-95c4-b889527cfa22-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:11.794339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.794173    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z9cc\" (UniqueName: \"kubernetes.io/projected/e886c2bd-b363-480c-95c4-b889527cfa22-kube-api-access-2z9cc\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:11.794339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.794188    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e886c2bd-b363-480c-95c4-b889527cfa22-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:11.798922 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.796302    5415 generic.go:356] "Generic (PLEG): container finished" podID="e886c2bd-b363-480c-95c4-b889527cfa22" containerID="2e194cd144ddb7a31575cd5e434233efa0b2e33cbd13aa6af49fe9a2d3d9b486" exitCode=0
Oct 31 04:28:11.798922 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.796375    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d" event={"ID":"e886c2bd-b363-480c-95c4-b889527cfa22","Type":"ContainerDied","Data":"2e194cd144ddb7a31575cd5e434233efa0b2e33cbd13aa6af49fe9a2d3d9b486"}
Oct 31 04:28:11.798922 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.796429    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d" event={"ID":"e886c2bd-b363-480c-95c4-b889527cfa22","Type":"ContainerDied","Data":"fd1e5d9dd555fca09c3009ab9bc7143dba38b81188773d07fe463321afa7e48b"}
Oct 31 04:28:11.798922 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.796461    5415 scope.go:122] "RemoveContainer" containerID="2e194cd144ddb7a31575cd5e434233efa0b2e33cbd13aa6af49fe9a2d3d9b486"
Oct 31 04:28:11.798922 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.796612    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"
Oct 31 04:28:11.812405 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.812379    5415 scope.go:122] "RemoveContainer" containerID="2e194cd144ddb7a31575cd5e434233efa0b2e33cbd13aa6af49fe9a2d3d9b486"
Oct 31 04:28:11.812809 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:28:11.812783    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e194cd144ddb7a31575cd5e434233efa0b2e33cbd13aa6af49fe9a2d3d9b486\": container with ID starting with 2e194cd144ddb7a31575cd5e434233efa0b2e33cbd13aa6af49fe9a2d3d9b486 not found: ID does not exist" containerID="2e194cd144ddb7a31575cd5e434233efa0b2e33cbd13aa6af49fe9a2d3d9b486"
Oct 31 04:28:11.812873 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.812826    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e194cd144ddb7a31575cd5e434233efa0b2e33cbd13aa6af49fe9a2d3d9b486"} err="failed to get container status \"2e194cd144ddb7a31575cd5e434233efa0b2e33cbd13aa6af49fe9a2d3d9b486\": rpc error: code = NotFound desc = could not find container \"2e194cd144ddb7a31575cd5e434233efa0b2e33cbd13aa6af49fe9a2d3d9b486\": container with ID starting with 2e194cd144ddb7a31575cd5e434233efa0b2e33cbd13aa6af49fe9a2d3d9b486 not found: ID does not exist"
Oct 31 04:28:11.956263 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.956176    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"]
Oct 31 04:28:11.959560 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:11.959530    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c0d50c-69b68bdb78-7d79d"]
Oct 31 04:28:12.604843 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:12.604800    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e886c2bd-b363-480c-95c4-b889527cfa22" path="/var/lib/kubelet/pods/e886c2bd-b363-480c-95c4-b889527cfa22/volumes"
Oct 31 04:28:16.175864 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:28:16.175828    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:28:20.706044 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:20.706001    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"]
Oct 31 04:28:29.210049 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:29.210008    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"]
Oct 31 04:28:35.726811 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:35.726772    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897" podUID="8164484e-3721-4998-bb76-278d2f4b5e35" containerName="main" containerID="cri-o://87e3d56414dbac717bd403826421dfb08daa52edc3d15e3c3c26f5f25ca19197" gracePeriod=45
Oct 31 04:28:35.926039 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:35.925993    5415 generic.go:356] "Generic (PLEG): container finished" podID="8164484e-3721-4998-bb76-278d2f4b5e35" containerID="87e3d56414dbac717bd403826421dfb08daa52edc3d15e3c3c26f5f25ca19197" exitCode=0
Oct 31 04:28:35.926218 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:35.926069    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897" event={"ID":"8164484e-3721-4998-bb76-278d2f4b5e35","Type":"ContainerDied","Data":"87e3d56414dbac717bd403826421dfb08daa52edc3d15e3c3c26f5f25ca19197"}
Oct 31 04:28:35.982701 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:35.982636    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 04:28:36.141280 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.141246    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-model-cache\") pod \"8164484e-3721-4998-bb76-278d2f4b5e35\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") "
Oct 31 04:28:36.141483 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.141306    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-dshm\") pod \"8164484e-3721-4998-bb76-278d2f4b5e35\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") "
Oct 31 04:28:36.141483 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.141365    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8164484e-3721-4998-bb76-278d2f4b5e35-kube-api-access-6q9s4\" (UniqueName: \"kubernetes.io/projected/8164484e-3721-4998-bb76-278d2f4b5e35-kube-api-access-6q9s4\") pod \"8164484e-3721-4998-bb76-278d2f4b5e35\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") "
Oct 31 04:28:36.141483 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.141407    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-tmp-dir\") pod \"8164484e-3721-4998-bb76-278d2f4b5e35\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") "
Oct 31 04:28:36.141483 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.141465    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-home\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-home\") pod \"8164484e-3721-4998-bb76-278d2f4b5e35\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") "
Oct 31 04:28:36.141483 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.141485    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8164484e-3721-4998-bb76-278d2f4b5e35-tls-certs\" (UniqueName: \"kubernetes.io/secret/8164484e-3721-4998-bb76-278d2f4b5e35-tls-certs\") pod \"8164484e-3721-4998-bb76-278d2f4b5e35\" (UID: \"8164484e-3721-4998-bb76-278d2f4b5e35\") "
Oct 31 04:28:36.141773 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.141612    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-model-cache" pod "8164484e-3721-4998-bb76-278d2f4b5e35" (UID: "8164484e-3721-4998-bb76-278d2f4b5e35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.141898 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.141847    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.142070 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.142044    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-tmp-dir" pod "8164484e-3721-4998-bb76-278d2f4b5e35" (UID: "8164484e-3721-4998-bb76-278d2f4b5e35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.142147 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.142064    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-home" pod "8164484e-3721-4998-bb76-278d2f4b5e35" (UID: "8164484e-3721-4998-bb76-278d2f4b5e35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.143642 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.143619    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-dshm" pod "8164484e-3721-4998-bb76-278d2f4b5e35" (UID: "8164484e-3721-4998-bb76-278d2f4b5e35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.143793 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.143770    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8164484e-3721-4998-bb76-278d2f4b5e35-tls-certs" pod "8164484e-3721-4998-bb76-278d2f4b5e35" (UID: "8164484e-3721-4998-bb76-278d2f4b5e35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:36.143865 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.143831    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8164484e-3721-4998-bb76-278d2f4b5e35-kube-api-access-6q9s4" pod "8164484e-3721-4998-bb76-278d2f4b5e35" (UID: "8164484e-3721-4998-bb76-278d2f4b5e35"). InnerVolumeSpecName "kube-api-access-6q9s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:36.242863 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.242776    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.242863 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.242809    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q9s4\" (UniqueName: \"kubernetes.io/projected/8164484e-3721-4998-bb76-278d2f4b5e35-kube-api-access-6q9s4\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.242863 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.242820    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.242863 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.242829    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8164484e-3721-4998-bb76-278d2f4b5e35-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.242863 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.242838    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8164484e-3721-4998-bb76-278d2f4b5e35-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.932832 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.932740    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897" event={"ID":"8164484e-3721-4998-bb76-278d2f4b5e35","Type":"ContainerDied","Data":"b7369da734a0800293fca9b28f69e7c5c4625b3992cfcfb10c0fbcf1b7148ab7"}
Oct 31 04:28:36.932832 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.932776    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"
Oct 31 04:28:36.932832 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:36.932797    5415 scope.go:122] "RemoveContainer" containerID="87e3d56414dbac717bd403826421dfb08daa52edc3d15e3c3c26f5f25ca19197"
Oct 31 04:28:37.091510 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:37.091471    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"]
Oct 31 04:28:37.138452 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:37.138420    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/distinct-w2-b59563-kserve-5f4d469db7-7w897"]
Oct 31 04:28:38.607996 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:38.607954    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8164484e-3721-4998-bb76-278d2f4b5e35" path="/var/lib/kubelet/pods/8164484e-3721-4998-bb76-278d2f4b5e35/volumes"
Oct 31 04:28:44.230858 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.230809    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" podUID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" containerName="main" containerID="cri-o://935c796299a40f1c605fff942b6224863a97b4673d614da3a537856b1ac6719f" gracePeriod=45
Oct 31 04:28:44.496212 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.496138    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 04:28:44.632503 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.632468    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tls-certs\" (UniqueName: \"kubernetes.io/secret/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tls-certs\") pod \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") "
Oct 31 04:28:44.632861 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.632528    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tmp-dir\") pod \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") "
Oct 31 04:28:44.632861 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.632575    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-kube-api-access-gfnkc\" (UniqueName: \"kubernetes.io/projected/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-kube-api-access-gfnkc\") pod \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") "
Oct 31 04:28:44.633653 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.633192    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-dshm\") pod \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") "
Oct 31 04:28:44.633653 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.633221    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tmp-dir" pod "72b65a20-6ba2-4f42-99e8-60d7a070aa5f" (UID: "72b65a20-6ba2-4f42-99e8-60d7a070aa5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.633653 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.633244    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-model-cache\") pod \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") "
Oct 31 04:28:44.633653 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.633355    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-home\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-home\") pod \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\" (UID: \"72b65a20-6ba2-4f42-99e8-60d7a070aa5f\") "
Oct 31 04:28:44.633894 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.633682    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-model-cache" pod "72b65a20-6ba2-4f42-99e8-60d7a070aa5f" (UID: "72b65a20-6ba2-4f42-99e8-60d7a070aa5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.633894 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.633738    5415 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tmp-dir\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.634009 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.633996    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-home" pod "72b65a20-6ba2-4f42-99e8-60d7a070aa5f" (UID: "72b65a20-6ba2-4f42-99e8-60d7a070aa5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.635414 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.635388    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tls-certs" pod "72b65a20-6ba2-4f42-99e8-60d7a070aa5f" (UID: "72b65a20-6ba2-4f42-99e8-60d7a070aa5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:44.635562 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.635532    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-kube-api-access-gfnkc" pod "72b65a20-6ba2-4f42-99e8-60d7a070aa5f" (UID: "72b65a20-6ba2-4f42-99e8-60d7a070aa5f"). InnerVolumeSpecName "kube-api-access-gfnkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:44.636989 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.636958    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-dshm" pod "72b65a20-6ba2-4f42-99e8-60d7a070aa5f" (UID: "72b65a20-6ba2-4f42-99e8-60d7a070aa5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.734661 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.734619    5415 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-dshm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.734661 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.734652    5415 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-model-cache\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.734661 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.734661    5415 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-home\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.734661 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.734670    5415 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-tls-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.734989 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.734679    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfnkc\" (UniqueName: \"kubernetes.io/projected/72b65a20-6ba2-4f42-99e8-60d7a070aa5f-kube-api-access-gfnkc\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.974628 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.974587    5415 generic.go:356] "Generic (PLEG): container finished" podID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" containerID="935c796299a40f1c605fff942b6224863a97b4673d614da3a537856b1ac6719f" exitCode=0
Oct 31 04:28:44.974628 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.974631    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"
Oct 31 04:28:44.974928 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.974670    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" event={"ID":"72b65a20-6ba2-4f42-99e8-60d7a070aa5f","Type":"ContainerDied","Data":"935c796299a40f1c605fff942b6224863a97b4673d614da3a537856b1ac6719f"}
Oct 31 04:28:44.974928 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.974723    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p" event={"ID":"72b65a20-6ba2-4f42-99e8-60d7a070aa5f","Type":"ContainerDied","Data":"9c3b907b4b97a630de28905e3b851f531d3548e2b958987d4bd40a10167ddc53"}
Oct 31 04:28:44.974928 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.974749    5415 scope.go:122] "RemoveContainer" containerID="935c796299a40f1c605fff942b6224863a97b4673d614da3a537856b1ac6719f"
Oct 31 04:28:44.984455 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.984432    5415 scope.go:122] "RemoveContainer" containerID="935c796299a40f1c605fff942b6224863a97b4673d614da3a537856b1ac6719f"
Oct 31 04:28:44.984767 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:28:44.984745    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935c796299a40f1c605fff942b6224863a97b4673d614da3a537856b1ac6719f\": container with ID starting with 935c796299a40f1c605fff942b6224863a97b4673d614da3a537856b1ac6719f not found: ID does not exist" containerID="935c796299a40f1c605fff942b6224863a97b4673d614da3a537856b1ac6719f"
Oct 31 04:28:44.984821 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:44.984777    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935c796299a40f1c605fff942b6224863a97b4673d614da3a537856b1ac6719f"} err="failed to get container status \"935c796299a40f1c605fff942b6224863a97b4673d614da3a537856b1ac6719f\": rpc error: code = NotFound desc = could not find container \"935c796299a40f1c605fff942b6224863a97b4673d614da3a537856b1ac6719f\": container with ID starting with 935c796299a40f1c605fff942b6224863a97b4673d614da3a537856b1ac6719f not found: ID does not exist"
Oct 31 04:28:45.138097 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:45.138013    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"]
Oct 31 04:28:45.184936 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:45.184889    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/distinct-w4-9360eb-kserve-5799546b88-t766p"]
Oct 31 04:28:46.175337 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:28:46.175298    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:28:46.604803 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:46.604770    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" path="/var/lib/kubelet/pods/72b65a20-6ba2-4f42-99e8-60d7a070aa5f/volumes"
Oct 31 04:28:49.402259 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.402218    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"]
Oct 31 04:28:49.402768 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.402485    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr" podUID="715cbcc7-1d2b-4fba-9527-4cf38503a4e3" containerName="maas-api" containerID="cri-o://0f3b2aa493c95487f9a77640fbf387e878a7e783f8c2a34ed63d52a9a6c95381" gracePeriod=30
Oct 31 04:28:49.665355 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.665276    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 04:28:49.773054 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.773019    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-openshift-service-ca\") pod \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") "
Oct 31 04:28:49.773238 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.773121    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-maas-api-tls\") pod \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") "
Oct 31 04:28:49.773238 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.773159    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-kube-api-access-bkvlg\" (UniqueName: \"kubernetes.io/projected/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-kube-api-access-bkvlg\") pod \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") "
Oct 31 04:28:49.773238 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.773186    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-metrics-certs\") pod \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\" (UID: \"715cbcc7-1d2b-4fba-9527-4cf38503a4e3\") "
Oct 31 04:28:49.773408 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.773384    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-openshift-service-ca" pod "715cbcc7-1d2b-4fba-9527-4cf38503a4e3" (UID: "715cbcc7-1d2b-4fba-9527-4cf38503a4e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:49.775620 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.775575    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-metrics-certs" pod "715cbcc7-1d2b-4fba-9527-4cf38503a4e3" (UID: "715cbcc7-1d2b-4fba-9527-4cf38503a4e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:49.775762 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.775685    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-kube-api-access-bkvlg" pod "715cbcc7-1d2b-4fba-9527-4cf38503a4e3" (UID: "715cbcc7-1d2b-4fba-9527-4cf38503a4e3"). InnerVolumeSpecName "kube-api-access-bkvlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:49.775762 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.775699    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-maas-api-tls" pod "715cbcc7-1d2b-4fba-9527-4cf38503a4e3" (UID: "715cbcc7-1d2b-4fba-9527-4cf38503a4e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:49.873984 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.873924    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-maas-api-tls\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:49.873984 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.873972    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkvlg\" (UniqueName: \"kubernetes.io/projected/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-kube-api-access-bkvlg\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:49.873984 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.873983    5415 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-metrics-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:49.874254 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:49.873999    5415 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715cbcc7-1d2b-4fba-9527-4cf38503a4e3-openshift-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:50.000501 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:50.000397    5415 generic.go:356] "Generic (PLEG): container finished" podID="715cbcc7-1d2b-4fba-9527-4cf38503a4e3" containerID="0f3b2aa493c95487f9a77640fbf387e878a7e783f8c2a34ed63d52a9a6c95381" exitCode=0
Oct 31 04:28:50.000501 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:50.000441    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"
Oct 31 04:28:50.000501 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:50.000481    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr" event={"ID":"715cbcc7-1d2b-4fba-9527-4cf38503a4e3","Type":"ContainerDied","Data":"0f3b2aa493c95487f9a77640fbf387e878a7e783f8c2a34ed63d52a9a6c95381"}
Oct 31 04:28:50.000823 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:50.000559    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr" event={"ID":"715cbcc7-1d2b-4fba-9527-4cf38503a4e3","Type":"ContainerDied","Data":"804395e669c68422af8c550f0bebe85e0c1c50d39cc1d818204654d11f90735d"}
Oct 31 04:28:50.000823 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:50.000593    5415 scope.go:122] "RemoveContainer" containerID="0f3b2aa493c95487f9a77640fbf387e878a7e783f8c2a34ed63d52a9a6c95381"
Oct 31 04:28:50.013422 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:50.013385    5415 scope.go:122] "RemoveContainer" containerID="0f3b2aa493c95487f9a77640fbf387e878a7e783f8c2a34ed63d52a9a6c95381"
Oct 31 04:28:50.013801 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:28:50.013781    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3b2aa493c95487f9a77640fbf387e878a7e783f8c2a34ed63d52a9a6c95381\": container with ID starting with 0f3b2aa493c95487f9a77640fbf387e878a7e783f8c2a34ed63d52a9a6c95381 not found: ID does not exist" containerID="0f3b2aa493c95487f9a77640fbf387e878a7e783f8c2a34ed63d52a9a6c95381"
Oct 31 04:28:50.013906 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:50.013815    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3b2aa493c95487f9a77640fbf387e878a7e783f8c2a34ed63d52a9a6c95381"} err="failed to get container status \"0f3b2aa493c95487f9a77640fbf387e878a7e783f8c2a34ed63d52a9a6c95381\": rpc error: code = NotFound desc = could not find container \"0f3b2aa493c95487f9a77640fbf387e878a7e783f8c2a34ed63d52a9a6c95381\": container with ID starting with 0f3b2aa493c95487f9a77640fbf387e878a7e783f8c2a34ed63d52a9a6c95381 not found: ID does not exist"
Oct 31 04:28:50.156700 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:50.156662    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"]
Oct 31 04:28:50.162199 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:50.162162    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9360eb-5884f99498-st7rr"]
Oct 31 04:28:50.605809 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:28:50.605771    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715cbcc7-1d2b-4fba-9527-4cf38503a4e3" path="/var/lib/kubelet/pods/715cbcc7-1d2b-4fba-9527-4cf38503a4e3/volumes"
Oct 31 04:29:01.990975 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:01.990933    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"]
Oct 31 04:29:01.991603 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:01.991154    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" podUID="dc949ce1-87f6-403b-a72d-1a517f979a65" containerName="istio-proxy" containerID="cri-o://a0a5738c01a7ace436549c325b7d092835038e77d9b77c90da25cf4503ff2ebe" gracePeriod=30
Oct 31 04:29:07.252572 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.252501    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 04:29:07.346903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.346836    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-kube-api-access-m78dr\" (UniqueName: \"kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-kube-api-access-m78dr\") pod \"dc949ce1-87f6-403b-a72d-1a517f979a65\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") "
Oct 31 04:29:07.346903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.346910    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-envoy\") pod \"dc949ce1-87f6-403b-a72d-1a517f979a65\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") "
Oct 31 04:29:07.347156 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.346951    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-certs\") pod \"dc949ce1-87f6-403b-a72d-1a517f979a65\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") "
Oct 31 04:29:07.347156 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.346979    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-socket\") pod \"dc949ce1-87f6-403b-a72d-1a517f979a65\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") "
Oct 31 04:29:07.347156 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.347057    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-credential-socket\") pod \"dc949ce1-87f6-403b-a72d-1a517f979a65\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") "
Oct 31 04:29:07.347156 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.347082    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-podinfo\") pod \"dc949ce1-87f6-403b-a72d-1a517f979a65\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") "
Oct 31 04:29:07.347156 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.347124    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-data\") pod \"dc949ce1-87f6-403b-a72d-1a517f979a65\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") "
Oct 31 04:29:07.347156 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.347151    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc949ce1-87f6-403b-a72d-1a517f979a65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc949ce1-87f6-403b-a72d-1a517f979a65-istiod-ca-cert\") pod \"dc949ce1-87f6-403b-a72d-1a517f979a65\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") "
Oct 31 04:29:07.347494 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.347179    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-token\" (UniqueName: \"kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-token\") pod \"dc949ce1-87f6-403b-a72d-1a517f979a65\" (UID: \"dc949ce1-87f6-403b-a72d-1a517f979a65\") "
Oct 31 04:29:07.347494 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.347292    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-certs" pod "dc949ce1-87f6-403b-a72d-1a517f979a65" (UID: "dc949ce1-87f6-403b-a72d-1a517f979a65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:07.347494 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.347334    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-socket" pod "dc949ce1-87f6-403b-a72d-1a517f979a65" (UID: "dc949ce1-87f6-403b-a72d-1a517f979a65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:07.347494 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.347380    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-credential-socket" pod "dc949ce1-87f6-403b-a72d-1a517f979a65" (UID: "dc949ce1-87f6-403b-a72d-1a517f979a65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:07.347718 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.347604    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:07.347718 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.347627    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:07.347718 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.347644    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:07.347897 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.347731    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-data" pod "dc949ce1-87f6-403b-a72d-1a517f979a65" (UID: "dc949ce1-87f6-403b-a72d-1a517f979a65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:07.347897 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.347766    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc949ce1-87f6-403b-a72d-1a517f979a65-istiod-ca-cert" pod "dc949ce1-87f6-403b-a72d-1a517f979a65" (UID: "dc949ce1-87f6-403b-a72d-1a517f979a65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:29:07.349278 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.349250    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-kube-api-access-m78dr" pod "dc949ce1-87f6-403b-a72d-1a517f979a65" (UID: "dc949ce1-87f6-403b-a72d-1a517f979a65"). InnerVolumeSpecName "kube-api-access-m78dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:29:07.349487 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.349465    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-podinfo" pod "dc949ce1-87f6-403b-a72d-1a517f979a65" (UID: "dc949ce1-87f6-403b-a72d-1a517f979a65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:29:07.349556 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.349474    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-envoy" pod "dc949ce1-87f6-403b-a72d-1a517f979a65" (UID: "dc949ce1-87f6-403b-a72d-1a517f979a65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:07.349556 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.349475    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-token" pod "dc949ce1-87f6-403b-a72d-1a517f979a65" (UID: "dc949ce1-87f6-403b-a72d-1a517f979a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:29:07.449390 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.449346    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:07.449390 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.449389    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc949ce1-87f6-403b-a72d-1a517f979a65-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:07.449675 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.449405    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:07.449675 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.449432    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m78dr\" (UniqueName: \"kubernetes.io/projected/dc949ce1-87f6-403b-a72d-1a517f979a65-kube-api-access-m78dr\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:07.449675 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.449449    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:07.449675 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:07.449463    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc949ce1-87f6-403b-a72d-1a517f979a65-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:08.087584 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:08.087548    5415 generic.go:356] "Generic (PLEG): container finished" podID="dc949ce1-87f6-403b-a72d-1a517f979a65" containerID="a0a5738c01a7ace436549c325b7d092835038e77d9b77c90da25cf4503ff2ebe" exitCode=0
Oct 31 04:29:08.087772 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:08.087599    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"
Oct 31 04:29:08.087772 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:08.087628    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" event={"ID":"dc949ce1-87f6-403b-a72d-1a517f979a65","Type":"ContainerDied","Data":"a0a5738c01a7ace436549c325b7d092835038e77d9b77c90da25cf4503ff2ebe"}
Oct 31 04:29:08.087772 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:08.087689    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94" event={"ID":"dc949ce1-87f6-403b-a72d-1a517f979a65","Type":"ContainerDied","Data":"3cd630d8bf5ba34bee3cd64667a6361f83b64446d3160507ba240daa4124aad7"}
Oct 31 04:29:08.087772 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:08.087711    5415 scope.go:122] "RemoveContainer" containerID="a0a5738c01a7ace436549c325b7d092835038e77d9b77c90da25cf4503ff2ebe"
Oct 31 04:29:08.097432 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:08.097401    5415 scope.go:122] "RemoveContainer" containerID="a0a5738c01a7ace436549c325b7d092835038e77d9b77c90da25cf4503ff2ebe"
Oct 31 04:29:08.097735 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:29:08.097714    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0a5738c01a7ace436549c325b7d092835038e77d9b77c90da25cf4503ff2ebe\": container with ID starting with a0a5738c01a7ace436549c325b7d092835038e77d9b77c90da25cf4503ff2ebe not found: ID does not exist" containerID="a0a5738c01a7ace436549c325b7d092835038e77d9b77c90da25cf4503ff2ebe"
Oct 31 04:29:08.097805 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:08.097746    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0a5738c01a7ace436549c325b7d092835038e77d9b77c90da25cf4503ff2ebe"} err="failed to get container status \"a0a5738c01a7ace436549c325b7d092835038e77d9b77c90da25cf4503ff2ebe\": rpc error: code = NotFound desc = could not find container \"a0a5738c01a7ace436549c325b7d092835038e77d9b77c90da25cf4503ff2ebe\": container with ID starting with a0a5738c01a7ace436549c325b7d092835038e77d9b77c90da25cf4503ff2ebe not found: ID does not exist"
Oct 31 04:29:08.241754 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:08.241718    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"]
Oct 31 04:29:08.288910 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:08.288859    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-l5n94"]
Oct 31 04:29:08.611061 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:08.611012    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc949ce1-87f6-403b-a72d-1a517f979a65" path="/var/lib/kubelet/pods/dc949ce1-87f6-403b-a72d-1a517f979a65/volumes"
Oct 31 04:29:10.076587 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:10.076550    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"]
Oct 31 04:29:10.077038 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:10.076788    5415 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm" podUID="fba7e0d3-2990-4db0-a832-b2d40e8fa461" containerName="istio-proxy" containerID="cri-o://01a246a3370a9f8912b8e7d5dc33a55ade51ce5a4a2f7e6b25159f2620943899" gracePeriod=30
Oct 31 04:29:12.280858 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:12.280828    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-f4q4d_b4e3fc57-e3fd-4a86-954c-70cf7062810f/kuadrant-console-plugin/0.log"
Oct 31 04:29:14.321905 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:14.321854    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nmdgv_6e19afe0-f05b-4294-8d01-8d04cbb5b833/manager/0.log"
Oct 31 04:29:15.334526 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.334497    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 04:29:15.424827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.424737    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-credential-socket\") pod \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") "
Oct 31 04:29:15.424827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.424773    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-envoy\") pod \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") "
Oct 31 04:29:15.424827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.424809    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-data\") pod \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") "
Oct 31 04:29:15.424827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.424826    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-socket\") pod \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") "
Oct 31 04:29:15.425234 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.424872    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-kube-api-access-xjzpm\" (UniqueName: \"kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-kube-api-access-xjzpm\") pod \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") "
Oct 31 04:29:15.425234 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.424951    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-certs\") pod \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") "
Oct 31 04:29:15.425234 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.425019    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-podinfo\") pod \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") "
Oct 31 04:29:15.425234 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.425074    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-credential-socket" pod "fba7e0d3-2990-4db0-a832-b2d40e8fa461" (UID: "fba7e0d3-2990-4db0-a832-b2d40e8fa461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:15.425234 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.425071    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istiod-ca-cert\") pod \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") "
Oct 31 04:29:15.425234 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.425153    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-token\" (UniqueName: \"kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-token\") pod \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\" (UID: \"fba7e0d3-2990-4db0-a832-b2d40e8fa461\") "
Oct 31 04:29:15.425234 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.425210    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-data" pod "fba7e0d3-2990-4db0-a832-b2d40e8fa461" (UID: "fba7e0d3-2990-4db0-a832-b2d40e8fa461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:15.425608 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.425424    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istiod-ca-cert" pod "fba7e0d3-2990-4db0-a832-b2d40e8fa461" (UID: "fba7e0d3-2990-4db0-a832-b2d40e8fa461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:29:15.425608 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.425445    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-certs" pod "fba7e0d3-2990-4db0-a832-b2d40e8fa461" (UID: "fba7e0d3-2990-4db0-a832-b2d40e8fa461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:15.425608 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.425579    5415 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-credential-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:15.425608 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.425600    5415 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-data\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:15.425766 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.425614    5415 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-certs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:15.425766 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.425628    5415 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istiod-ca-cert\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:15.425766 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.425629    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-socket" pod "fba7e0d3-2990-4db0-a832-b2d40e8fa461" (UID: "fba7e0d3-2990-4db0-a832-b2d40e8fa461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:15.427240 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.427214    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-kube-api-access-xjzpm" pod "fba7e0d3-2990-4db0-a832-b2d40e8fa461" (UID: "fba7e0d3-2990-4db0-a832-b2d40e8fa461"). InnerVolumeSpecName "kube-api-access-xjzpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:29:15.427340 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.427236    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-envoy" pod "fba7e0d3-2990-4db0-a832-b2d40e8fa461" (UID: "fba7e0d3-2990-4db0-a832-b2d40e8fa461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:15.427597 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.427579    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-podinfo" pod "fba7e0d3-2990-4db0-a832-b2d40e8fa461" (UID: "fba7e0d3-2990-4db0-a832-b2d40e8fa461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:29:15.427646 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.427592    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-token" pod "fba7e0d3-2990-4db0-a832-b2d40e8fa461" (UID: "fba7e0d3-2990-4db0-a832-b2d40e8fa461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:29:15.526678 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.526644    5415 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-token\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:15.526678 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.526671    5415 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-envoy\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:15.526678 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.526685    5415 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba7e0d3-2990-4db0-a832-b2d40e8fa461-workload-socket\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:15.526956 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.526699    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjzpm\" (UniqueName: \"kubernetes.io/projected/fba7e0d3-2990-4db0-a832-b2d40e8fa461-kube-api-access-xjzpm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:15.526956 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.526710    5415 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba7e0d3-2990-4db0-a832-b2d40e8fa461-istio-podinfo\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:15.708762 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:15.708673    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qwnr5_81594aa6-a125-4f16-9c62-5b3b6f24c6fb/manager/0.log"
Oct 31 04:29:16.128729 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:16.128697    5415 generic.go:356] "Generic (PLEG): container finished" podID="fba7e0d3-2990-4db0-a832-b2d40e8fa461" containerID="01a246a3370a9f8912b8e7d5dc33a55ade51ce5a4a2f7e6b25159f2620943899" exitCode=0
Oct 31 04:29:16.128962 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:16.128739    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"
Oct 31 04:29:16.128962 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:16.128762    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm" event={"ID":"fba7e0d3-2990-4db0-a832-b2d40e8fa461","Type":"ContainerDied","Data":"01a246a3370a9f8912b8e7d5dc33a55ade51ce5a4a2f7e6b25159f2620943899"}
Oct 31 04:29:16.128962 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:16.128816    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm" event={"ID":"fba7e0d3-2990-4db0-a832-b2d40e8fa461","Type":"ContainerDied","Data":"55eb208b243a103e857609bd43a2517e2f25e9995e4bd3f306076eff0de2d303"}
Oct 31 04:29:16.128962 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:16.128845    5415 scope.go:122] "RemoveContainer" containerID="01a246a3370a9f8912b8e7d5dc33a55ade51ce5a4a2f7e6b25159f2620943899"
Oct 31 04:29:16.138345 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:16.138321    5415 scope.go:122] "RemoveContainer" containerID="01a246a3370a9f8912b8e7d5dc33a55ade51ce5a4a2f7e6b25159f2620943899"
Oct 31 04:29:16.138639 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:29:16.138619    5415 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a246a3370a9f8912b8e7d5dc33a55ade51ce5a4a2f7e6b25159f2620943899\": container with ID starting with 01a246a3370a9f8912b8e7d5dc33a55ade51ce5a4a2f7e6b25159f2620943899 not found: ID does not exist" containerID="01a246a3370a9f8912b8e7d5dc33a55ade51ce5a4a2f7e6b25159f2620943899"
Oct 31 04:29:16.138752 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:16.138649    5415 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a246a3370a9f8912b8e7d5dc33a55ade51ce5a4a2f7e6b25159f2620943899"} err="failed to get container status \"01a246a3370a9f8912b8e7d5dc33a55ade51ce5a4a2f7e6b25159f2620943899\": rpc error: code = NotFound desc = could not find container \"01a246a3370a9f8912b8e7d5dc33a55ade51ce5a4a2f7e6b25159f2620943899\": container with ID starting with 01a246a3370a9f8912b8e7d5dc33a55ade51ce5a4a2f7e6b25159f2620943899 not found: ID does not exist"
Oct 31 04:29:16.175432 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:29:16.175397    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:29:16.187288 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:16.187257    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cmm9t_e94e222c-4de2-4046-a659-0c92420cc03e/manager/0.log"
Oct 31 04:29:16.329895 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:16.329855    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"]
Oct 31 04:29:16.376836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:16.376797    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mf5tm"]
Oct 31 04:29:16.605494 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:16.605460    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba7e0d3-2990-4db0-a832-b2d40e8fa461" path="/var/lib/kubelet/pods/fba7e0d3-2990-4db0-a832-b2d40e8fa461/volumes"
Oct 31 04:29:16.883864 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:16.883780    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4m5hw_35e1e5d7-2240-4292-9f78-a9676f67e28a/server/0.log"
Oct 31 04:29:17.108155 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:17.108103    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hpjht_6cb52f07-5218-4b28-896c-9b97e062ef44/manager/0.log"
Oct 31 04:29:17.377473 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:29:17.377444    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-bk6x5_9dcba361-a1d8-4e62-843c-be420bd3fea8/proxy/0.log"
Oct 31 04:29:46.174659 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:29:46.174620    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:30:00.151731 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.151694    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"]
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152066    5415 cpu_manager.go:414] "containerMap: removing container" podUID="fba7e0d3-2990-4db0-a832-b2d40e8fa461" containerName="istio-proxy"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152078    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba7e0d3-2990-4db0-a832-b2d40e8fa461" containerName="istio-proxy"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152093    5415 cpu_manager.go:414] "containerMap: removing container" podUID="e8a80930-06e4-4b4e-9aec-79274f0961f0" containerName="main"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152098    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a80930-06e4-4b4e-9aec-79274f0961f0" containerName="main"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152115    5415 cpu_manager.go:414] "containerMap: removing container" podUID="dc949ce1-87f6-403b-a72d-1a517f979a65" containerName="istio-proxy"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152121    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc949ce1-87f6-403b-a72d-1a517f979a65" containerName="istio-proxy"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152127    5415 cpu_manager.go:414] "containerMap: removing container" podUID="abc2f709-4ac0-40c9-bea6-5306df98321c" containerName="maas-api"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152132    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc2f709-4ac0-40c9-bea6-5306df98321c" containerName="maas-api"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152138    5415 cpu_manager.go:414] "containerMap: removing container" podUID="35e6897c-0910-4afe-bdbb-7a19250f6735" containerName="payload-pre-processing"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152145    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e6897c-0910-4afe-bdbb-7a19250f6735" containerName="payload-pre-processing"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152153    5415 cpu_manager.go:414] "containerMap: removing container" podUID="2ad27d6d-957f-4a3b-bcef-5a1bab901463" containerName="maas-api"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152158    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad27d6d-957f-4a3b-bcef-5a1bab901463" containerName="maas-api"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152164    5415 cpu_manager.go:414] "containerMap: removing container" podUID="013cbda1-7eb1-4fbe-8e03-c2de7b71928a" containerName="istio-proxy"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152169    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013cbda1-7eb1-4fbe-8e03-c2de7b71928a" containerName="istio-proxy"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152180    5415 cpu_manager.go:414] "containerMap: removing container" podUID="7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" containerName="main"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152184    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" containerName="main"
Oct 31 04:30:00.152180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152191    5415 cpu_manager.go:414] "containerMap: removing container" podUID="7286f18c-2456-47f7-a04d-078677b37a27" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152196    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7286f18c-2456-47f7-a04d-078677b37a27" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152201    5415 cpu_manager.go:414] "containerMap: removing container" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152206    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152211    5415 cpu_manager.go:414] "containerMap: removing container" podUID="d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" containerName="istio-proxy"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152217    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" containerName="istio-proxy"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152225    5415 cpu_manager.go:414] "containerMap: removing container" podUID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152229    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152237    5415 cpu_manager.go:414] "containerMap: removing container" podUID="f39ffc5b-0fbf-41ff-91c9-9242538ed564" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152241    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f39ffc5b-0fbf-41ff-91c9-9242538ed564" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152248    5415 cpu_manager.go:414] "containerMap: removing container" podUID="187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152252    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152259    5415 cpu_manager.go:414] "containerMap: removing container" podUID="8164484e-3721-4998-bb76-278d2f4b5e35" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152264    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8164484e-3721-4998-bb76-278d2f4b5e35" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152272    5415 cpu_manager.go:414] "containerMap: removing container" podUID="e886c2bd-b363-480c-95c4-b889527cfa22" containerName="maas-api"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152276    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e886c2bd-b363-480c-95c4-b889527cfa22" containerName="maas-api"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152283    5415 cpu_manager.go:414] "containerMap: removing container" podUID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" containerName="istio-proxy"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152287    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" containerName="istio-proxy"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152293    5415 cpu_manager.go:414] "containerMap: removing container" podUID="37142a22-1f4a-4bbe-ac5b-72a3ff276832" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152298    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37142a22-1f4a-4bbe-ac5b-72a3ff276832" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152307    5415 cpu_manager.go:414] "containerMap: removing container" podUID="b54532f2-51ef-4ade-a041-b3c0c74105bf" containerName="maas-api"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152311    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b54532f2-51ef-4ade-a041-b3c0c74105bf" containerName="maas-api"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152318    5415 cpu_manager.go:414] "containerMap: removing container" podUID="715cbcc7-1d2b-4fba-9527-4cf38503a4e3" containerName="maas-api"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152323    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715cbcc7-1d2b-4fba-9527-4cf38503a4e3" containerName="maas-api"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152370    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="37142a22-1f4a-4bbe-ac5b-72a3ff276832" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152378    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad27d6d-957f-4a3b-bcef-5a1bab901463" containerName="maas-api"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152384    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="013cbda1-7eb1-4fbe-8e03-c2de7b71928a" containerName="istio-proxy"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152391    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="715cbcc7-1d2b-4fba-9527-4cf38503a4e3" containerName="maas-api"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152396    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="b54532f2-51ef-4ade-a041-b3c0c74105bf" containerName="maas-api"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152402    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1f3a9b-f45a-44ab-9ee7-8c91aa066f5a" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152407    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="7286f18c-2456-47f7-a04d-078677b37a27" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152412    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1ad969-4b4b-4178-a5a9-7f76d8cd2150" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152418    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba7e0d3-2990-4db0-a832-b2d40e8fa461" containerName="istio-proxy"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152423    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03920f9-a5dc-4eb0-8e03-4f5fb85f80a1" containerName="istio-proxy"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152428    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b65a20-6ba2-4f42-99e8-60d7a070aa5f" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152433    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a80930-06e4-4b4e-9aec-79274f0961f0" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152438    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="f39ffc5b-0fbf-41ff-91c9-9242538ed564" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152445    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="8164484e-3721-4998-bb76-278d2f4b5e35" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152450    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="187dfdc0-69f1-4a06-b4c3-ccae0eeb6d51" containerName="main"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152455    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="e886c2bd-b363-480c-95c4-b889527cfa22" containerName="maas-api"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152460    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc949ce1-87f6-403b-a72d-1a517f979a65" containerName="istio-proxy"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152464    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc2f709-4ac0-40c9-bea6-5306df98321c" containerName="maas-api"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152470    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a09ce1-e619-406a-8097-ea1c5b8cbd3e" containerName="istio-proxy"
Oct 31 04:30:00.152807 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.152475    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e6897c-0910-4afe-bdbb-7a19250f6735" containerName="payload-pre-processing"
Oct 31 04:30:00.155531 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.155512    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"
Oct 31 04:30:00.205198 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.205162    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:30:00.232641 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.232600    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bltsm\" (UniqueName: \"kubernetes.io/projected/dc2e6f04-4243-4a8b-b124-032e2fd42c10-kube-api-access-bltsm\") pod \"maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc\" (UID: \"dc2e6f04-4243-4a8b-b124-032e2fd42c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"
Oct 31 04:30:00.232809 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.232657    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e6f04-4243-4a8b-b124-032e2fd42c10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc\" (UID: \"dc2e6f04-4243-4a8b-b124-032e2fd42c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"
Oct 31 04:30:00.253865 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.253830    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"]
Oct 31 04:30:00.333422 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.333382    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e6f04-4243-4a8b-b124-032e2fd42c10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc\" (UID: \"dc2e6f04-4243-4a8b-b124-032e2fd42c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"
Oct 31 04:30:00.333645 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.333502    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bltsm\" (UniqueName: \"kubernetes.io/projected/dc2e6f04-4243-4a8b-b124-032e2fd42c10-kube-api-access-bltsm\") pod \"maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc\" (UID: \"dc2e6f04-4243-4a8b-b124-032e2fd42c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"
Oct 31 04:30:00.334125 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.334100    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e6f04-4243-4a8b-b124-032e2fd42c10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc\" (UID: \"dc2e6f04-4243-4a8b-b124-032e2fd42c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"
Oct 31 04:30:00.387820 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.387740    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bltsm\" (UniqueName: \"kubernetes.io/projected/dc2e6f04-4243-4a8b-b124-032e2fd42c10-kube-api-access-bltsm\") pod \"maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc\" (UID: \"dc2e6f04-4243-4a8b-b124-032e2fd42c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"
Oct 31 04:30:00.468726 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.468679    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"
Oct 31 04:30:00.831959 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:00.831868    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"]
Oct 31 04:30:00.834957 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:30:00.834920    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2e6f04_4243_4a8b_b124_032e2fd42c10.slice/crio-4e3ec157960e43b343cd90693c3e0d9b0cdad888586fb7e1d7eeaaabd120bbc1 WatchSource:0}: Error finding container 4e3ec157960e43b343cd90693c3e0d9b0cdad888586fb7e1d7eeaaabd120bbc1: Status 404 returned error can't find the container with id 4e3ec157960e43b343cd90693c3e0d9b0cdad888586fb7e1d7eeaaabd120bbc1
Oct 31 04:30:01.339616 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:01.339576    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc" event={"ID":"dc2e6f04-4243-4a8b-b124-032e2fd42c10","Type":"ContainerStarted","Data":"0114e63c4e16a9b70b3c9cdb07396b0a77267bb59e4e793e85e982323d7055ea"}
Oct 31 04:30:01.340041 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:01.339641    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc" event={"ID":"dc2e6f04-4243-4a8b-b124-032e2fd42c10","Type":"ContainerStarted","Data":"4e3ec157960e43b343cd90693c3e0d9b0cdad888586fb7e1d7eeaaabd120bbc1"}
Oct 31 04:30:01.446774 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:01.446683    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc" podStartSLOduration=1.446669712 podStartE2EDuration="1.446669712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:01.446056178 +0000 UTC m=+4943.006535910" watchObservedRunningTime="2026-10-31 04:30:01.446669712 +0000 UTC m=+4943.007149442"
Oct 31 04:30:02.353123 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:02.353087    5415 generic.go:356] "Generic (PLEG): container finished" podID="dc2e6f04-4243-4a8b-b124-032e2fd42c10" containerID="0114e63c4e16a9b70b3c9cdb07396b0a77267bb59e4e793e85e982323d7055ea" exitCode=60
Oct 31 04:30:02.353539 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:02.353170    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc" event={"ID":"dc2e6f04-4243-4a8b-b124-032e2fd42c10","Type":"ContainerDied","Data":"0114e63c4e16a9b70b3c9cdb07396b0a77267bb59e4e793e85e982323d7055ea"}
Oct 31 04:30:02.353539 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:02.353389    5415 scope.go:122] "RemoveContainer" containerID="0114e63c4e16a9b70b3c9cdb07396b0a77267bb59e4e793e85e982323d7055ea"
Oct 31 04:30:03.359973 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:03.359933    5415 generic.go:356] "Generic (PLEG): container finished" podID="dc2e6f04-4243-4a8b-b124-032e2fd42c10" containerID="5818c6948952b07bc03c591a0802275a528d0b3e3f069f66f393e70ac8e39f1d" exitCode=60
Oct 31 04:30:03.360437 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:03.360010    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc" event={"ID":"dc2e6f04-4243-4a8b-b124-032e2fd42c10","Type":"ContainerDied","Data":"5818c6948952b07bc03c591a0802275a528d0b3e3f069f66f393e70ac8e39f1d"}
Oct 31 04:30:03.360437 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:03.360064    5415 scope.go:122] "RemoveContainer" containerID="0114e63c4e16a9b70b3c9cdb07396b0a77267bb59e4e793e85e982323d7055ea"
Oct 31 04:30:03.360437 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:03.360309    5415 scope.go:122] "RemoveContainer" containerID="5818c6948952b07bc03c591a0802275a528d0b3e3f069f66f393e70ac8e39f1d"
Oct 31 04:30:03.360580 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:30:03.360528    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc_odh-ai-gateway-infra(dc2e6f04-4243-4a8b-b124-032e2fd42c10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc" podUID="dc2e6f04-4243-4a8b-b124-032e2fd42c10"
Oct 31 04:30:04.366981 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:04.366951    5415 scope.go:122] "RemoveContainer" containerID="5818c6948952b07bc03c591a0802275a528d0b3e3f069f66f393e70ac8e39f1d"
Oct 31 04:30:04.367398 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:30:04.367176    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc_odh-ai-gateway-infra(dc2e6f04-4243-4a8b-b124-032e2fd42c10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc" podUID="dc2e6f04-4243-4a8b-b124-032e2fd42c10"
Oct 31 04:30:13.598547 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:13.598509    5415 scope.go:122] "RemoveContainer" containerID="5818c6948952b07bc03c591a0802275a528d0b3e3f069f66f393e70ac8e39f1d"
Oct 31 04:30:14.412363 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:14.412329    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc" event={"ID":"dc2e6f04-4243-4a8b-b124-032e2fd42c10","Type":"ContainerStarted","Data":"b580f4b26632e1d80777deec48736c2279d18da2fac97063492ce33d09e323f7"}
Oct 31 04:30:15.420013 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:15.419925    5415 generic.go:356] "Generic (PLEG): container finished" podID="dc2e6f04-4243-4a8b-b124-032e2fd42c10" containerID="b580f4b26632e1d80777deec48736c2279d18da2fac97063492ce33d09e323f7" exitCode=60
Oct 31 04:30:15.420013 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:15.419975    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc" event={"ID":"dc2e6f04-4243-4a8b-b124-032e2fd42c10","Type":"ContainerDied","Data":"b580f4b26632e1d80777deec48736c2279d18da2fac97063492ce33d09e323f7"}
Oct 31 04:30:15.420521 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:15.420028    5415 scope.go:122] "RemoveContainer" containerID="5818c6948952b07bc03c591a0802275a528d0b3e3f069f66f393e70ac8e39f1d"
Oct 31 04:30:15.420521 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:15.420406    5415 scope.go:122] "RemoveContainer" containerID="b580f4b26632e1d80777deec48736c2279d18da2fac97063492ce33d09e323f7"
Oct 31 04:30:15.420709 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:30:15.420684    5415 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc_odh-ai-gateway-infra(dc2e6f04-4243-4a8b-b124-032e2fd42c10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc" podUID="dc2e6f04-4243-4a8b-b124-032e2fd42c10"
Oct 31 04:30:15.527613 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:15.527577    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"]
Oct 31 04:30:16.176021 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:30:16.175987    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:30:16.563692 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:16.563663    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"
Oct 31 04:30:16.686559 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:16.686475    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2e6f04-4243-4a8b-b124-032e2fd42c10-kube-api-access-bltsm\" (UniqueName: \"kubernetes.io/projected/dc2e6f04-4243-4a8b-b124-032e2fd42c10-kube-api-access-bltsm\") pod \"dc2e6f04-4243-4a8b-b124-032e2fd42c10\" (UID: \"dc2e6f04-4243-4a8b-b124-032e2fd42c10\") "
Oct 31 04:30:16.686559 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:16.686531    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc2e6f04-4243-4a8b-b124-032e2fd42c10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e6f04-4243-4a8b-b124-032e2fd42c10-maas-api-service-ca\") pod \"dc2e6f04-4243-4a8b-b124-032e2fd42c10\" (UID: \"dc2e6f04-4243-4a8b-b124-032e2fd42c10\") "
Oct 31 04:30:16.686956 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:16.686933    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2e6f04-4243-4a8b-b124-032e2fd42c10-maas-api-service-ca" pod "dc2e6f04-4243-4a8b-b124-032e2fd42c10" (UID: "dc2e6f04-4243-4a8b-b124-032e2fd42c10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:16.688825 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:16.688797    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2e6f04-4243-4a8b-b124-032e2fd42c10-kube-api-access-bltsm" pod "dc2e6f04-4243-4a8b-b124-032e2fd42c10" (UID: "dc2e6f04-4243-4a8b-b124-032e2fd42c10"). InnerVolumeSpecName "kube-api-access-bltsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:16.787663 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:16.787627    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bltsm\" (UniqueName: \"kubernetes.io/projected/dc2e6f04-4243-4a8b-b124-032e2fd42c10-kube-api-access-bltsm\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:16.787663 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:16.787656    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e6f04-4243-4a8b-b124-032e2fd42c10-maas-api-service-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:17.432835 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:17.432805    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"
Oct 31 04:30:17.433189 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:17.432801    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc" event={"ID":"dc2e6f04-4243-4a8b-b124-032e2fd42c10","Type":"ContainerDied","Data":"4e3ec157960e43b343cd90693c3e0d9b0cdad888586fb7e1d7eeaaabd120bbc1"}
Oct 31 04:30:17.433189 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:17.432948    5415 scope.go:122] "RemoveContainer" containerID="b580f4b26632e1d80777deec48736c2279d18da2fac97063492ce33d09e323f7"
Oct 31 04:30:17.588625 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:17.588589    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"]
Oct 31 04:30:17.635699 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:17.635663    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e863e405-29890350-qpgpc"]
Oct 31 04:30:18.605654 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:30:18.605620    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc2e6f04-4243-4a8b-b124-032e2fd42c10" path="/var/lib/kubelet/pods/dc2e6f04-4243-4a8b-b124-032e2fd42c10/volumes"
Oct 31 04:30:46.174470 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:30:46.174427    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:31:07.680227 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.680187    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"]
Oct 31 04:31:07.680688 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.680578    5415 cpu_manager.go:414] "containerMap: removing container" podUID="dc2e6f04-4243-4a8b-b124-032e2fd42c10" containerName="cleanup"
Oct 31 04:31:07.680688 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.680591    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2e6f04-4243-4a8b-b124-032e2fd42c10" containerName="cleanup"
Oct 31 04:31:07.680688 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.680604    5415 cpu_manager.go:414] "containerMap: removing container" podUID="dc2e6f04-4243-4a8b-b124-032e2fd42c10" containerName="cleanup"
Oct 31 04:31:07.680688 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.680610    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2e6f04-4243-4a8b-b124-032e2fd42c10" containerName="cleanup"
Oct 31 04:31:07.680688 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.680683    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2e6f04-4243-4a8b-b124-032e2fd42c10" containerName="cleanup"
Oct 31 04:31:07.680855 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.680694    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2e6f04-4243-4a8b-b124-032e2fd42c10" containerName="cleanup"
Oct 31 04:31:07.683948 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.683925    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.858082 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.858043    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.858082 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.858082    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtg57\" (UniqueName: \"kubernetes.io/projected/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-kube-api-access-xtg57\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.858303 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.858176    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.858303 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.858226    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.858380 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.858315    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.858380 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.858346    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.858444 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.858427    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.858479 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.858444    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.858479 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.858462    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.895363 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.895326    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"]
Oct 31 04:31:07.959640 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.959550    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.959640 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.959591    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.959640 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.959629    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.959640 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.959643    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.960075 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.959662    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.960075 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.959693    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.960075 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.959711    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtg57\" (UniqueName: \"kubernetes.io/projected/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-kube-api-access-xtg57\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.960075 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.959765    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.960075 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.959803    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.960357 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.960076    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.960357 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.960188    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.960357 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.960241    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.960357 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.960334    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.960555 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.960459    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.962104 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.962081    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:07.962239 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:07.962220    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:08.013244 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:08.013200    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:08.013494 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:08.013469    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtg57\" (UniqueName: \"kubernetes.io/projected/c0469dfd-14dc-458a-9b3e-d5b0ecc25000-kube-api-access-xtg57\") pod \"maas-default-gateway-openshift-default-6886b96f9-wlt76\" (UID: \"c0469dfd-14dc-458a-9b3e-d5b0ecc25000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:08.297720 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:08.297614    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:08.715198 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:08.715175    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"]
Oct 31 04:31:08.718145 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:31:08.718113    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0469dfd_14dc_458a_9b3e_d5b0ecc25000.slice/crio-9b31f1b264350660182715eaa63a93ce669f96d0e27984a371b015083e036d2b WatchSource:0}: Error finding container 9b31f1b264350660182715eaa63a93ce669f96d0e27984a371b015083e036d2b: Status 404 returned error can't find the container with id 9b31f1b264350660182715eaa63a93ce669f96d0e27984a371b015083e036d2b
Oct 31 04:31:08.720478 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:08.720446    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:31:08.720570 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:08.720515    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:31:08.720570 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:08.720548    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:31:09.676763 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:09.676723    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76" event={"ID":"c0469dfd-14dc-458a-9b3e-d5b0ecc25000","Type":"ContainerStarted","Data":"62d6bd3a71f0a9c6dc1aa6666f8641864d5099af4cc1e4b4c2633c8bfb9e8e36"}
Oct 31 04:31:09.676990 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:09.676787    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76" event={"ID":"c0469dfd-14dc-458a-9b3e-d5b0ecc25000","Type":"ContainerStarted","Data":"9b31f1b264350660182715eaa63a93ce669f96d0e27984a371b015083e036d2b"}
Oct 31 04:31:09.836062 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:09.836006    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76" podStartSLOduration=2.835991112 podStartE2EDuration="2.835991112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:09.83499441 +0000 UTC m=+5011.395474162" watchObservedRunningTime="2026-10-31 04:31:09.835991112 +0000 UTC m=+5011.396470843"
Oct 31 04:31:10.298171 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:10.298127    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:10.299633 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:10.299608    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wlt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 04:31:10.299690 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:10.299660    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76" podUID="c0469dfd-14dc-458a-9b3e-d5b0ecc25000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 04:31:11.298855 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:11.298815    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wlt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 04:31:11.299299 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:11.298901    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76" podUID="c0469dfd-14dc-458a-9b3e-d5b0ecc25000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 04:31:12.298619 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:12.298572    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wlt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 04:31:12.298838 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:12.298636    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76" podUID="c0469dfd-14dc-458a-9b3e-d5b0ecc25000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 04:31:13.298121 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:13.298079    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wlt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 04:31:13.298567 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:13.298152    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76" podUID="c0469dfd-14dc-458a-9b3e-d5b0ecc25000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 04:31:14.297958 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:14.297925    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wlt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 04:31:14.298183 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:14.297983    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76" podUID="c0469dfd-14dc-458a-9b3e-d5b0ecc25000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 04:31:15.298452 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:15.298406    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wlt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 04:31:15.298896 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:15.298480    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76" podUID="c0469dfd-14dc-458a-9b3e-d5b0ecc25000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 04:31:16.175196 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:31:16.175140    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:31:17.038107 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:17.038075    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:17.038629 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:17.038512    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:17.039442 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:31:17.039422    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76"
Oct 31 04:31:46.175227 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:31:46.175146    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:32:07.680301 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.680261    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"]
Oct 31 04:32:07.680916 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.680861    5415 cpu_manager.go:414] "containerMap: removing container" podUID="dc2e6f04-4243-4a8b-b124-032e2fd42c10" containerName="cleanup"
Oct 31 04:32:07.680964 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.680920    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2e6f04-4243-4a8b-b124-032e2fd42c10" containerName="cleanup"
Oct 31 04:32:07.681272 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.681034    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2e6f04-4243-4a8b-b124-032e2fd42c10" containerName="cleanup"
Oct 31 04:32:07.684533 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.684510    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.787431 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.787392    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"]
Oct 31 04:32:07.844615 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.844572    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.844615 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.844618    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.844944 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.844715    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.844944 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.844778    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.844944 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.844804    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9pjd\" (UniqueName: \"kubernetes.io/projected/1cfc6a64-edfa-4e51-83b5-f38f438f1877-kube-api-access-h9pjd\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.844944 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.844921    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.845165 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.844963    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.845165 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.845059    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.845165 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.845103    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.945661 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.945576    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.945661 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.945626    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.945661 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.945661    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.945981 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.945691    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.945981 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.945730    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.946328 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.946122    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.946328 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.946184    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.946328 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.946205    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.946328 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.946191    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.946328 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.946255    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.946328 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.946282    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pjd\" (UniqueName: \"kubernetes.io/projected/1cfc6a64-edfa-4e51-83b5-f38f438f1877-kube-api-access-h9pjd\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.946774 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.946635    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.946774 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.946710    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.947081 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.947060    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.948143 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.948121    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:07.948414 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:07.948397    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:08.000703 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:08.000675    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cfc6a64-edfa-4e51-83b5-f38f438f1877-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:08.000703 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:08.000697    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pjd\" (UniqueName: \"kubernetes.io/projected/1cfc6a64-edfa-4e51-83b5-f38f438f1877-kube-api-access-h9pjd\") pod \"maas-default-gateway-openshift-default-6886b96f9-5smg5\" (UID: \"1cfc6a64-edfa-4e51-83b5-f38f438f1877\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:08.298093 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:08.297984    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:08.713491 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:08.713460    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"]
Oct 31 04:32:08.716393 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:32:08.716361    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfc6a64_edfa_4e51_83b5_f38f438f1877.slice/crio-081293e5e65f3f4ff22cb754a5e9c6b10144684ac6eeedd5959fea0457a7972c WatchSource:0}: Error finding container 081293e5e65f3f4ff22cb754a5e9c6b10144684ac6eeedd5959fea0457a7972c: Status 404 returned error can't find the container with id 081293e5e65f3f4ff22cb754a5e9c6b10144684ac6eeedd5959fea0457a7972c
Oct 31 04:32:08.718454 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:08.718424    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:32:08.718537 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:08.718492    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:32:08.718537 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:08.718521    5415 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 04:32:08.955443 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:08.955405    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" event={"ID":"1cfc6a64-edfa-4e51-83b5-f38f438f1877","Type":"ContainerStarted","Data":"0b3770a04b42bee2d3afc8830cdbe4caa29cd9b988669a8fe1dd691b715a78db"}
Oct 31 04:32:08.955633 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:08.955507    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" event={"ID":"1cfc6a64-edfa-4e51-83b5-f38f438f1877","Type":"ContainerStarted","Data":"081293e5e65f3f4ff22cb754a5e9c6b10144684ac6eeedd5959fea0457a7972c"}
Oct 31 04:32:09.065908 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:09.065763    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" podStartSLOduration=2.065745835 podStartE2EDuration="2.065745835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:09.065642266 +0000 UTC m=+5070.626122023" watchObservedRunningTime="2026-10-31 04:32:09.065745835 +0000 UTC m=+5070.626225564"
Oct 31 04:32:09.298125 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:09.298091    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:09.299769 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:09.299740    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5smg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 31 04:32:09.299933 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:09.299806    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" podUID="1cfc6a64-edfa-4e51-83b5-f38f438f1877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 31 04:32:10.298351 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:10.298310    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5smg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 31 04:32:10.298904 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:10.298366    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" podUID="1cfc6a64-edfa-4e51-83b5-f38f438f1877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 31 04:32:11.298489 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:11.298441    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5smg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 31 04:32:11.298939 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:11.298506    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" podUID="1cfc6a64-edfa-4e51-83b5-f38f438f1877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 31 04:32:12.298693 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:12.298652    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5smg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 31 04:32:12.299147 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:12.298719    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" podUID="1cfc6a64-edfa-4e51-83b5-f38f438f1877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 31 04:32:13.302422 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:13.302377    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5smg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 31 04:32:13.303094 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:13.303057    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" podUID="1cfc6a64-edfa-4e51-83b5-f38f438f1877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 31 04:32:14.301902 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:14.301255    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5smg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 31 04:32:14.301902 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:14.301412    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" podUID="1cfc6a64-edfa-4e51-83b5-f38f438f1877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 31 04:32:15.298760 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:15.298701    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5smg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 31 04:32:15.299371 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:15.298796    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" podUID="1cfc6a64-edfa-4e51-83b5-f38f438f1877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 31 04:32:16.175522 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:32:16.175480    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:32:16.298833 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:16.298797    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5smg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 31 04:32:16.299268 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:16.298867    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" podUID="1cfc6a64-edfa-4e51-83b5-f38f438f1877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 31 04:32:18.341067 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:18.341032    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5smg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:32:18.341595 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:18.341134    5415 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" podUID="1cfc6a64-edfa-4e51-83b5-f38f438f1877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:32:19.197442 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:19.197398    5415 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:19.197809 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:19.197786    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:19.198652 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:32:19.198635    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5"
Oct 31 04:32:46.174235 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:32:46.174195    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:33:16.235886 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:33:16.235787    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:33:46.175320 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:33:46.175277    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:34:05.092364 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.092326    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"]
Oct 31 04:34:05.095960 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.095934    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"
Oct 31 04:34:05.192124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.192088    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:34:05.192124 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.192089    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:34:05.193463 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.193440    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"]
Oct 31 04:34:05.217455 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.217413    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e37e06ba-cwn2l\" (UID: \"cb0c4754-1518-42de-9632-b1eeffa51f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"
Oct 31 04:34:05.217668 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.217577    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e37e06ba-cwn2l\" (UID: \"cb0c4754-1518-42de-9632-b1eeffa51f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"
Oct 31 04:34:05.217668 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.217656    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vtw6\" (UniqueName: \"kubernetes.io/projected/cb0c4754-1518-42de-9632-b1eeffa51f55-kube-api-access-5vtw6\") pod \"maas-api-revoke-sub-e37e06ba-cwn2l\" (UID: \"cb0c4754-1518-42de-9632-b1eeffa51f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"
Oct 31 04:34:05.318800 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.318760    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e37e06ba-cwn2l\" (UID: \"cb0c4754-1518-42de-9632-b1eeffa51f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"
Oct 31 04:34:05.319022 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.318818    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vtw6\" (UniqueName: \"kubernetes.io/projected/cb0c4754-1518-42de-9632-b1eeffa51f55-kube-api-access-5vtw6\") pod \"maas-api-revoke-sub-e37e06ba-cwn2l\" (UID: \"cb0c4754-1518-42de-9632-b1eeffa51f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"
Oct 31 04:34:05.319022 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.318934    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e37e06ba-cwn2l\" (UID: \"cb0c4754-1518-42de-9632-b1eeffa51f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"
Oct 31 04:34:05.319526 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.319501    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e37e06ba-cwn2l\" (UID: \"cb0c4754-1518-42de-9632-b1eeffa51f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"
Oct 31 04:34:05.321350 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.321328    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e37e06ba-cwn2l\" (UID: \"cb0c4754-1518-42de-9632-b1eeffa51f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"
Oct 31 04:34:05.372959 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.372868    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vtw6\" (UniqueName: \"kubernetes.io/projected/cb0c4754-1518-42de-9632-b1eeffa51f55-kube-api-access-5vtw6\") pod \"maas-api-revoke-sub-e37e06ba-cwn2l\" (UID: \"cb0c4754-1518-42de-9632-b1eeffa51f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"
Oct 31 04:34:05.408567 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.408527    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"
Oct 31 04:34:05.771757 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:05.771681    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"]
Oct 31 04:34:05.775323 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:34:05.775296    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0c4754_1518_42de_9632_b1eeffa51f55.slice/crio-6e9635967e45a17aae48e5930c42a7a5eb162f2d12a3e605c46c39c99c293949 WatchSource:0}: Error finding container 6e9635967e45a17aae48e5930c42a7a5eb162f2d12a3e605c46c39c99c293949: Status 404 returned error can't find the container with id 6e9635967e45a17aae48e5930c42a7a5eb162f2d12a3e605c46c39c99c293949
Oct 31 04:34:06.583492 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:06.583461    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l" event={"ID":"cb0c4754-1518-42de-9632-b1eeffa51f55","Type":"ContainerStarted","Data":"47b8149c9ab29b545ac6e7b01e04c056353ef4067afebeb1d6eac55ba5fa5322"}
Oct 31 04:34:06.583888 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:06.583516    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l" event={"ID":"cb0c4754-1518-42de-9632-b1eeffa51f55","Type":"ContainerStarted","Data":"6e9635967e45a17aae48e5930c42a7a5eb162f2d12a3e605c46c39c99c293949"}
Oct 31 04:34:06.691671 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:06.691617    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l" podStartSLOduration=1.691600182 podStartE2EDuration="1.691600182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:06.691366029 +0000 UTC m=+5188.251845773" watchObservedRunningTime="2026-10-31 04:34:06.691600182 +0000 UTC m=+5188.252079916"
Oct 31 04:34:07.590074 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:07.590036    5415 generic.go:356] "Generic (PLEG): container finished" podID="cb0c4754-1518-42de-9632-b1eeffa51f55" containerID="47b8149c9ab29b545ac6e7b01e04c056353ef4067afebeb1d6eac55ba5fa5322" exitCode=0
Oct 31 04:34:07.590607 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:07.590111    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l" event={"ID":"cb0c4754-1518-42de-9632-b1eeffa51f55","Type":"ContainerDied","Data":"47b8149c9ab29b545ac6e7b01e04c056353ef4067afebeb1d6eac55ba5fa5322"}
Oct 31 04:34:08.738914 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:08.738872    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"
Oct 31 04:34:08.851530 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:08.851439    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb0c4754-1518-42de-9632-b1eeffa51f55-kube-api-access-5vtw6\" (UniqueName: \"kubernetes.io/projected/cb0c4754-1518-42de-9632-b1eeffa51f55-kube-api-access-5vtw6\") pod \"cb0c4754-1518-42de-9632-b1eeffa51f55\" (UID: \"cb0c4754-1518-42de-9632-b1eeffa51f55\") "
Oct 31 04:34:08.851712 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:08.851605    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-openshift-ca\") pod \"cb0c4754-1518-42de-9632-b1eeffa51f55\" (UID: \"cb0c4754-1518-42de-9632-b1eeffa51f55\") "
Oct 31 04:34:08.851712 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:08.851655    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-xks-ca\") pod \"cb0c4754-1518-42de-9632-b1eeffa51f55\" (UID: \"cb0c4754-1518-42de-9632-b1eeffa51f55\") "
Oct 31 04:34:08.852047 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:08.852022    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-openshift-ca" pod "cb0c4754-1518-42de-9632-b1eeffa51f55" (UID: "cb0c4754-1518-42de-9632-b1eeffa51f55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:34:08.854199 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:08.854171    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-xks-ca" pod "cb0c4754-1518-42de-9632-b1eeffa51f55" (UID: "cb0c4754-1518-42de-9632-b1eeffa51f55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:34:08.856247 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:08.856220    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0c4754-1518-42de-9632-b1eeffa51f55-kube-api-access-5vtw6" pod "cb0c4754-1518-42de-9632-b1eeffa51f55" (UID: "cb0c4754-1518-42de-9632-b1eeffa51f55"). InnerVolumeSpecName "kube-api-access-5vtw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:34:08.952800 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:08.952760    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:34:08.952800 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:08.952793    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0c4754-1518-42de-9632-b1eeffa51f55-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:34:08.952800 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:08.952804    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vtw6\" (UniqueName: \"kubernetes.io/projected/cb0c4754-1518-42de-9632-b1eeffa51f55-kube-api-access-5vtw6\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:34:09.601740 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:09.601707    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l" event={"ID":"cb0c4754-1518-42de-9632-b1eeffa51f55","Type":"ContainerDied","Data":"6e9635967e45a17aae48e5930c42a7a5eb162f2d12a3e605c46c39c99c293949"}
Oct 31 04:34:09.601740 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:09.601737    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"
Oct 31 04:34:09.601990 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:09.601754    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e9635967e45a17aae48e5930c42a7a5eb162f2d12a3e605c46c39c99c293949"
Oct 31 04:34:16.175172 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:34:16.175132    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:34:23.584214 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.584170    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"]
Oct 31 04:34:23.584800 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.584692    5415 cpu_manager.go:414] "containerMap: removing container" podUID="cb0c4754-1518-42de-9632-b1eeffa51f55" containerName="revoke-keys"
Oct 31 04:34:23.584800 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.584709    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0c4754-1518-42de-9632-b1eeffa51f55" containerName="revoke-keys"
Oct 31 04:34:23.584800 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.584795    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0c4754-1518-42de-9632-b1eeffa51f55" containerName="revoke-keys"
Oct 31 04:34:23.588100 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.588070    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"
Oct 31 04:34:23.638094 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.638048    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:34:23.685081 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.685020    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:34:23.687435 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.687400    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"]
Oct 31 04:34:23.695424 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.695389    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5554fcc-xvbxw\" (UID: \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"
Oct 31 04:34:23.695659 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.695626    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5554fcc-xvbxw\" (UID: \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"
Oct 31 04:34:23.695761 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.695668    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czk24\" (UniqueName: \"kubernetes.io/projected/cfa7f077-805c-4e84-8d77-02a9c58a11a5-kube-api-access-czk24\") pod \"maas-api-revoke-sub-b5554fcc-xvbxw\" (UID: \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"
Oct 31 04:34:23.796751 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.796700    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5554fcc-xvbxw\" (UID: \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"
Oct 31 04:34:23.796978 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.796845    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5554fcc-xvbxw\" (UID: \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"
Oct 31 04:34:23.796978 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.796906    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czk24\" (UniqueName: \"kubernetes.io/projected/cfa7f077-805c-4e84-8d77-02a9c58a11a5-kube-api-access-czk24\") pod \"maas-api-revoke-sub-b5554fcc-xvbxw\" (UID: \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"
Oct 31 04:34:23.797955 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.797917    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5554fcc-xvbxw\" (UID: \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"
Oct 31 04:34:23.800004 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.799976    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5554fcc-xvbxw\" (UID: \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"
Oct 31 04:34:23.851785 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.851711    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czk24\" (UniqueName: \"kubernetes.io/projected/cfa7f077-805c-4e84-8d77-02a9c58a11a5-kube-api-access-czk24\") pod \"maas-api-revoke-sub-b5554fcc-xvbxw\" (UID: \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"
Oct 31 04:34:23.906503 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:23.906466    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"
Oct 31 04:34:24.289852 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:24.289758    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"]
Oct 31 04:34:24.695901 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:24.695527    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw" event={"ID":"cfa7f077-805c-4e84-8d77-02a9c58a11a5","Type":"ContainerStarted","Data":"1b14dc972f7d9c6416d515ad3a65e46834440b806cda567af906fd878a58ddd8"}
Oct 31 04:34:25.715452 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:25.715307    5415 generic.go:356] "Generic (PLEG): container finished" podID="cfa7f077-805c-4e84-8d77-02a9c58a11a5" containerID="d7192463f3dfb0025c765e02d035dadb79eea5d645166ca28184614ba14fb8a0" exitCode=0
Oct 31 04:34:25.715452 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:25.715401    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw" event={"ID":"cfa7f077-805c-4e84-8d77-02a9c58a11a5","Type":"ContainerDied","Data":"d7192463f3dfb0025c765e02d035dadb79eea5d645166ca28184614ba14fb8a0"}
Oct 31 04:34:26.853790 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:26.853761    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"
Oct 31 04:34:27.043163 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:27.043080    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa7f077-805c-4e84-8d77-02a9c58a11a5-kube-api-access-czk24\" (UniqueName: \"kubernetes.io/projected/cfa7f077-805c-4e84-8d77-02a9c58a11a5-kube-api-access-czk24\") pod \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\" (UID: \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\") "
Oct 31 04:34:27.043163 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:27.043144    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-xks-ca\") pod \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\" (UID: \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\") "
Oct 31 04:34:27.043361 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:27.043270    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-openshift-ca\") pod \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\" (UID: \"cfa7f077-805c-4e84-8d77-02a9c58a11a5\") "
Oct 31 04:34:27.043656 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:27.043625    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-openshift-ca" pod "cfa7f077-805c-4e84-8d77-02a9c58a11a5" (UID: "cfa7f077-805c-4e84-8d77-02a9c58a11a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:34:27.045167 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:27.045147    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-xks-ca" pod "cfa7f077-805c-4e84-8d77-02a9c58a11a5" (UID: "cfa7f077-805c-4e84-8d77-02a9c58a11a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:34:27.045315 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:27.045295    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa7f077-805c-4e84-8d77-02a9c58a11a5-kube-api-access-czk24" pod "cfa7f077-805c-4e84-8d77-02a9c58a11a5" (UID: "cfa7f077-805c-4e84-8d77-02a9c58a11a5"). InnerVolumeSpecName "kube-api-access-czk24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:34:27.144002 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:27.143944    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:34:27.144002 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:27.143993    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czk24\" (UniqueName: \"kubernetes.io/projected/cfa7f077-805c-4e84-8d77-02a9c58a11a5-kube-api-access-czk24\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:34:27.144002 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:27.144007    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa7f077-805c-4e84-8d77-02a9c58a11a5-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:34:27.726967 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:27.726940    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"
Oct 31 04:34:27.726967 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:27.726958    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw" event={"ID":"cfa7f077-805c-4e84-8d77-02a9c58a11a5","Type":"ContainerDied","Data":"1b14dc972f7d9c6416d515ad3a65e46834440b806cda567af906fd878a58ddd8"}
Oct 31 04:34:27.727199 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:34:27.727013    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b14dc972f7d9c6416d515ad3a65e46834440b806cda567af906fd878a58ddd8"
Oct 31 04:34:46.174751 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:34:46.174666    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:35:16.174769 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:35:16.174728    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:35:46.175470 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:35:46.175431    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:36:16.175422 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:36:16.175335    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:36:46.174582 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:36:46.174540    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:37:16.174135 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:37:16.174091    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:37:46.175518 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:37:46.175382    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:38:16.174848 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:38:16.174797    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:38:46.174443 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:38:46.174399    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:39:09.049092 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:39:09.049056    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"]
Oct 31 04:39:09.095783 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:39:09.095733    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e37e06ba-cwn2l"]
Oct 31 04:39:10.604613 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:39:10.604578    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb0c4754-1518-42de-9632-b1eeffa51f55" path="/var/lib/kubelet/pods/cb0c4754-1518-42de-9632-b1eeffa51f55/volumes"
Oct 31 04:39:16.174335 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:39:16.174253    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:39:27.043353 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:39:27.043314    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"]
Oct 31 04:39:27.090420 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:39:27.090381    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5554fcc-xvbxw"]
Oct 31 04:39:28.604648 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:39:28.604617    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa7f077-805c-4e84-8d77-02a9c58a11a5" path="/var/lib/kubelet/pods/cfa7f077-805c-4e84-8d77-02a9c58a11a5/volumes"
Oct 31 04:39:46.175380 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:39:46.175343    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:40:16.175236 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:40:16.175199    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:40:40.031268 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:40:40.031238    5415 scope.go:122] "RemoveContainer" containerID="d7192463f3dfb0025c765e02d035dadb79eea5d645166ca28184614ba14fb8a0"
Oct 31 04:40:40.039786 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:40:40.039764    5415 scope.go:122] "RemoveContainer" containerID="47b8149c9ab29b545ac6e7b01e04c056353ef4067afebeb1d6eac55ba5fa5322"
Oct 31 04:40:46.175399 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:40:46.175316    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:41:16.175976 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:41:16.175935    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:41:46.174251 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:41:46.174200    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:41:48.092902 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:41:48.091944    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wlt76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:41:48.092902 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:41:48.092024    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76" podUID="c0469dfd-14dc-458a-9b3e-d5b0ecc25000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:42:16.174854 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:42:16.174764    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:42:46.173861 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:42:46.173813    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:43:16.173326 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:43:16.173287    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:43:46.174569 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:43:46.174480    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:44:16.176648 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:44:16.176610    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:44:46.174636 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:44:46.174596    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:45:01.008418 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.008371    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"]
Oct 31 04:45:01.009943 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.009914    5415 cpu_manager.go:414] "containerMap: removing container" podUID="cfa7f077-805c-4e84-8d77-02a9c58a11a5" containerName="revoke-keys"
Oct 31 04:45:01.010143 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.010126    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa7f077-805c-4e84-8d77-02a9c58a11a5" containerName="revoke-keys"
Oct 31 04:45:01.010473 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.010450    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa7f077-805c-4e84-8d77-02a9c58a11a5" containerName="revoke-keys"
Oct 31 04:45:01.015631 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.015606    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"
Oct 31 04:45:01.075903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.067509    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"]
Oct 31 04:45:01.115467 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.115430    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:45:01.115793 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.115773    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:45:01.126708 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.126671    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp69w\" (UniqueName: \"kubernetes.io/projected/4fd871dc-2354-409e-90e8-0d1d855c864d-kube-api-access-dp69w\") pod \"maas-api-revoke-sub-0b7e8edb-g8z6z\" (UID: \"4fd871dc-2354-409e-90e8-0d1d855c864d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"
Oct 31 04:45:01.127297 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.127243    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-g8z6z\" (UID: \"4fd871dc-2354-409e-90e8-0d1d855c864d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"
Oct 31 04:45:01.127729 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.127708    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-g8z6z\" (UID: \"4fd871dc-2354-409e-90e8-0d1d855c864d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"
Oct 31 04:45:01.228726 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.228691    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-g8z6z\" (UID: \"4fd871dc-2354-409e-90e8-0d1d855c864d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"
Oct 31 04:45:01.228939 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.228761    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-g8z6z\" (UID: \"4fd871dc-2354-409e-90e8-0d1d855c864d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"
Oct 31 04:45:01.228939 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.228811    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp69w\" (UniqueName: \"kubernetes.io/projected/4fd871dc-2354-409e-90e8-0d1d855c864d-kube-api-access-dp69w\") pod \"maas-api-revoke-sub-0b7e8edb-g8z6z\" (UID: \"4fd871dc-2354-409e-90e8-0d1d855c864d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"
Oct 31 04:45:01.229579 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.229553    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-g8z6z\" (UID: \"4fd871dc-2354-409e-90e8-0d1d855c864d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"
Oct 31 04:45:01.231369 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.231336    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-g8z6z\" (UID: \"4fd871dc-2354-409e-90e8-0d1d855c864d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"
Oct 31 04:45:01.313952 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.313067    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp69w\" (UniqueName: \"kubernetes.io/projected/4fd871dc-2354-409e-90e8-0d1d855c864d-kube-api-access-dp69w\") pod \"maas-api-revoke-sub-0b7e8edb-g8z6z\" (UID: \"4fd871dc-2354-409e-90e8-0d1d855c864d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"
Oct 31 04:45:01.333257 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.333200    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"
Oct 31 04:45:01.834311 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.834276    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"]
Oct 31 04:45:01.841278 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:45:01.841246    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd871dc_2354_409e_90e8_0d1d855c864d.slice/crio-a0bfffd4669a9aac75df4ace7aa7919afc1da741080b59694f24919d29ed7348 WatchSource:0}: Error finding container a0bfffd4669a9aac75df4ace7aa7919afc1da741080b59694f24919d29ed7348: Status 404 returned error can't find the container with id a0bfffd4669a9aac75df4ace7aa7919afc1da741080b59694f24919d29ed7348
Oct 31 04:45:01.932271 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:01.932231    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z" event={"ID":"4fd871dc-2354-409e-90e8-0d1d855c864d","Type":"ContainerStarted","Data":"a0bfffd4669a9aac75df4ace7aa7919afc1da741080b59694f24919d29ed7348"}
Oct 31 04:45:02.938888 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:02.938854    5415 generic.go:356] "Generic (PLEG): container finished" podID="4fd871dc-2354-409e-90e8-0d1d855c864d" containerID="bb650b819cfc5f4455623074fde379c19548d35aec446a7666bf3817d5802774" exitCode=0
Oct 31 04:45:02.939386 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:02.938952    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z" event={"ID":"4fd871dc-2354-409e-90e8-0d1d855c864d","Type":"ContainerDied","Data":"bb650b819cfc5f4455623074fde379c19548d35aec446a7666bf3817d5802774"}
Oct 31 04:45:04.080050 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:04.080018    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"
Oct 31 04:45:04.163768 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:04.163731    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-openshift-ca\") pod \"4fd871dc-2354-409e-90e8-0d1d855c864d\" (UID: \"4fd871dc-2354-409e-90e8-0d1d855c864d\") "
Oct 31 04:45:04.163985 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:04.163787    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd871dc-2354-409e-90e8-0d1d855c864d-kube-api-access-dp69w\" (UniqueName: \"kubernetes.io/projected/4fd871dc-2354-409e-90e8-0d1d855c864d-kube-api-access-dp69w\") pod \"4fd871dc-2354-409e-90e8-0d1d855c864d\" (UID: \"4fd871dc-2354-409e-90e8-0d1d855c864d\") "
Oct 31 04:45:04.163985 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:04.163824    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-xks-ca\") pod \"4fd871dc-2354-409e-90e8-0d1d855c864d\" (UID: \"4fd871dc-2354-409e-90e8-0d1d855c864d\") "
Oct 31 04:45:04.164152 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:04.164130    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-openshift-ca" pod "4fd871dc-2354-409e-90e8-0d1d855c864d" (UID: "4fd871dc-2354-409e-90e8-0d1d855c864d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:45:04.166089 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:04.166056    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-xks-ca" pod "4fd871dc-2354-409e-90e8-0d1d855c864d" (UID: "4fd871dc-2354-409e-90e8-0d1d855c864d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:45:04.166089 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:04.166072    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd871dc-2354-409e-90e8-0d1d855c864d-kube-api-access-dp69w" pod "4fd871dc-2354-409e-90e8-0d1d855c864d" (UID: "4fd871dc-2354-409e-90e8-0d1d855c864d"). InnerVolumeSpecName "kube-api-access-dp69w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:45:04.265629 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:04.265535    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:45:04.265629 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:04.265568    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp69w\" (UniqueName: \"kubernetes.io/projected/4fd871dc-2354-409e-90e8-0d1d855c864d-kube-api-access-dp69w\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:45:04.265629 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:04.265578    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd871dc-2354-409e-90e8-0d1d855c864d-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:45:04.950658 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:04.950579    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"
Oct 31 04:45:04.950821 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:04.950572    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z" event={"ID":"4fd871dc-2354-409e-90e8-0d1d855c864d","Type":"ContainerDied","Data":"a0bfffd4669a9aac75df4ace7aa7919afc1da741080b59694f24919d29ed7348"}
Oct 31 04:45:04.950821 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:04.950716    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0bfffd4669a9aac75df4ace7aa7919afc1da741080b59694f24919d29ed7348"
Oct 31 04:45:12.196046 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.196009    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"]
Oct 31 04:45:12.196537 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.196427    5415 cpu_manager.go:414] "containerMap: removing container" podUID="4fd871dc-2354-409e-90e8-0d1d855c864d" containerName="revoke-keys"
Oct 31 04:45:12.196537 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.196438    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd871dc-2354-409e-90e8-0d1d855c864d" containerName="revoke-keys"
Oct 31 04:45:12.196537 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.196520    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd871dc-2354-409e-90e8-0d1d855c864d" containerName="revoke-keys"
Oct 31 04:45:12.200002 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.199974    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"
Oct 31 04:45:12.249500 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.249467    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:45:12.296562 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.296515    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:45:12.297996 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.297971    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"]
Oct 31 04:45:12.340648 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.340600    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwhgl\" (UniqueName: \"kubernetes.io/projected/5621931c-d489-4316-8d4b-4d8c923cb1da-kube-api-access-fwhgl\") pod \"maas-api-revoke-sub-3d4156f1-qn4hk\" (UID: \"5621931c-d489-4316-8d4b-4d8c923cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"
Oct 31 04:45:12.340835 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.340782    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qn4hk\" (UID: \"5621931c-d489-4316-8d4b-4d8c923cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"
Oct 31 04:45:12.340835 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.340813    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qn4hk\" (UID: \"5621931c-d489-4316-8d4b-4d8c923cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"
Oct 31 04:45:12.441934 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.441852    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qn4hk\" (UID: \"5621931c-d489-4316-8d4b-4d8c923cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"
Oct 31 04:45:12.442151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.441949    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qn4hk\" (UID: \"5621931c-d489-4316-8d4b-4d8c923cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"
Oct 31 04:45:12.442151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.442008    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwhgl\" (UniqueName: \"kubernetes.io/projected/5621931c-d489-4316-8d4b-4d8c923cb1da-kube-api-access-fwhgl\") pod \"maas-api-revoke-sub-3d4156f1-qn4hk\" (UID: \"5621931c-d489-4316-8d4b-4d8c923cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"
Oct 31 04:45:12.442621 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.442595    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qn4hk\" (UID: \"5621931c-d489-4316-8d4b-4d8c923cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"
Oct 31 04:45:12.444352 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.444327    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qn4hk\" (UID: \"5621931c-d489-4316-8d4b-4d8c923cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"
Oct 31 04:45:12.495490 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.495417    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwhgl\" (UniqueName: \"kubernetes.io/projected/5621931c-d489-4316-8d4b-4d8c923cb1da-kube-api-access-fwhgl\") pod \"maas-api-revoke-sub-3d4156f1-qn4hk\" (UID: \"5621931c-d489-4316-8d4b-4d8c923cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"
Oct 31 04:45:12.513357 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.513325    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"
Oct 31 04:45:12.878943 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.878918    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"]
Oct 31 04:45:12.881900 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:45:12.881842    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5621931c_d489_4316_8d4b_4d8c923cb1da.slice/crio-1279ae987e03bae9306f3c7d99e05ccbab664cb260111ed48ad83280e1148d05 WatchSource:0}: Error finding container 1279ae987e03bae9306f3c7d99e05ccbab664cb260111ed48ad83280e1148d05: Status 404 returned error can't find the container with id 1279ae987e03bae9306f3c7d99e05ccbab664cb260111ed48ad83280e1148d05
Oct 31 04:45:12.987296 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:12.987258    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk" event={"ID":"5621931c-d489-4316-8d4b-4d8c923cb1da","Type":"ContainerStarted","Data":"1279ae987e03bae9306f3c7d99e05ccbab664cb260111ed48ad83280e1148d05"}
Oct 31 04:45:13.994352 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:13.994311    5415 generic.go:356] "Generic (PLEG): container finished" podID="5621931c-d489-4316-8d4b-4d8c923cb1da" containerID="bd4bf9abf528be07156707d237694e71a362239b73ae8b3d9568f1165a32e66c" exitCode=0
Oct 31 04:45:13.994771 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:13.994397    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk" event={"ID":"5621931c-d489-4316-8d4b-4d8c923cb1da","Type":"ContainerDied","Data":"bd4bf9abf528be07156707d237694e71a362239b73ae8b3d9568f1165a32e66c"}
Oct 31 04:45:15.138778 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:15.138751    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"
Oct 31 04:45:15.268728 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:15.268644    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-openshift-ca\") pod \"5621931c-d489-4316-8d4b-4d8c923cb1da\" (UID: \"5621931c-d489-4316-8d4b-4d8c923cb1da\") "
Oct 31 04:45:15.268921 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:15.268737    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-xks-ca\") pod \"5621931c-d489-4316-8d4b-4d8c923cb1da\" (UID: \"5621931c-d489-4316-8d4b-4d8c923cb1da\") "
Oct 31 04:45:15.268921 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:15.268840    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5621931c-d489-4316-8d4b-4d8c923cb1da-kube-api-access-fwhgl\" (UniqueName: \"kubernetes.io/projected/5621931c-d489-4316-8d4b-4d8c923cb1da-kube-api-access-fwhgl\") pod \"5621931c-d489-4316-8d4b-4d8c923cb1da\" (UID: \"5621931c-d489-4316-8d4b-4d8c923cb1da\") "
Oct 31 04:45:15.269170 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:15.269141    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-openshift-ca" pod "5621931c-d489-4316-8d4b-4d8c923cb1da" (UID: "5621931c-d489-4316-8d4b-4d8c923cb1da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:45:15.269282 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:15.269262    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:45:15.271037 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:15.271008    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-xks-ca" pod "5621931c-d489-4316-8d4b-4d8c923cb1da" (UID: "5621931c-d489-4316-8d4b-4d8c923cb1da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:45:15.271037 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:15.271010    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5621931c-d489-4316-8d4b-4d8c923cb1da-kube-api-access-fwhgl" pod "5621931c-d489-4316-8d4b-4d8c923cb1da" (UID: "5621931c-d489-4316-8d4b-4d8c923cb1da"). InnerVolumeSpecName "kube-api-access-fwhgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:45:15.370458 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:15.370413    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwhgl\" (UniqueName: \"kubernetes.io/projected/5621931c-d489-4316-8d4b-4d8c923cb1da-kube-api-access-fwhgl\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:45:15.370458 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:15.370450    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5621931c-d489-4316-8d4b-4d8c923cb1da-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:45:16.005549 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:16.005522    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"
Oct 31 04:45:16.005549 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:16.005541    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk" event={"ID":"5621931c-d489-4316-8d4b-4d8c923cb1da","Type":"ContainerDied","Data":"1279ae987e03bae9306f3c7d99e05ccbab664cb260111ed48ad83280e1148d05"}
Oct 31 04:45:16.005775 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:16.005586    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1279ae987e03bae9306f3c7d99e05ccbab664cb260111ed48ad83280e1148d05"
Oct 31 04:45:16.176354 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:45:16.176312    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:45:35.241112 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:35.241065    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5smg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:45:35.241603 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:35.241144    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" podUID="1cfc6a64-edfa-4e51-83b5-f38f438f1877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:45:38.430164 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:38.430123    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"]
Oct 31 04:45:38.477061 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:38.477018    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qn4hk"]
Oct 31 04:45:38.605303 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:45:38.605272    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5621931c-d489-4316-8d4b-4d8c923cb1da" path="/var/lib/kubelet/pods/5621931c-d489-4316-8d4b-4d8c923cb1da/volumes"
Oct 31 04:45:46.174828 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:45:46.174791    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:46:16.177674 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:46:16.177635    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:46:35.241095 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:46:35.241051    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5smg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:46:35.241605 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:46:35.241122    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" podUID="1cfc6a64-edfa-4e51-83b5-f38f438f1877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:46:46.175299 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:46:46.175218    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:47:16.174638 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:47:16.174601    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:47:46.175023 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:47:46.174986    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:48:16.175410 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:48:16.175274    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:48:46.175157 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:48:46.175112    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:49:16.176549 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:49:16.176506    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:49:46.174612 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:49:46.174528    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:50:05.043454 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:50:05.043414    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"]
Oct 31 04:50:05.090371 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:50:05.090336    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-g8z6z"]
Oct 31 04:50:06.604922 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:50:06.604868    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd871dc-2354-409e-90e8-0d1d855c864d" path="/var/lib/kubelet/pods/4fd871dc-2354-409e-90e8-0d1d855c864d/volumes"
Oct 31 04:50:16.173926 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:50:16.173867    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:50:46.174189 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:50:46.174151    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:51:16.176468 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:51:16.176382    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:51:40.134337 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:51:40.134306    5415 scope.go:122] "RemoveContainer" containerID="bd4bf9abf528be07156707d237694e71a362239b73ae8b3d9568f1165a32e66c"
Oct 31 04:51:40.143029 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:51:40.143001    5415 scope.go:122] "RemoveContainer" containerID="bb650b819cfc5f4455623074fde379c19548d35aec446a7666bf3817d5802774"
Oct 31 04:51:46.175572 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:51:46.175534    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:52:16.175406 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:52:16.175369    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:52:16.341157 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.341118    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"]
Oct 31 04:52:16.341553 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.341540    5415 cpu_manager.go:414] "containerMap: removing container" podUID="5621931c-d489-4316-8d4b-4d8c923cb1da" containerName="revoke-keys"
Oct 31 04:52:16.341599 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.341554    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5621931c-d489-4316-8d4b-4d8c923cb1da" containerName="revoke-keys"
Oct 31 04:52:16.341642 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.341633    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="5621931c-d489-4316-8d4b-4d8c923cb1da" containerName="revoke-keys"
Oct 31 04:52:16.344889 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.344856    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"
Oct 31 04:52:16.394149 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.394115    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:52:16.441287 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.441195    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:52:16.442562 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.442542    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"]
Oct 31 04:52:16.446583 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.446558    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbxhs\" (UniqueName: \"kubernetes.io/projected/68271017-34eb-41ed-a4c9-d9d3db251d7f-kube-api-access-zbxhs\") pod \"maas-api-revoke-sub-f0fb7551-sj26p\" (UID: \"68271017-34eb-41ed-a4c9-d9d3db251d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"
Oct 31 04:52:16.446723 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.446698    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fb7551-sj26p\" (UID: \"68271017-34eb-41ed-a4c9-d9d3db251d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"
Oct 31 04:52:16.446795 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.446764    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fb7551-sj26p\" (UID: \"68271017-34eb-41ed-a4c9-d9d3db251d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"
Oct 31 04:52:16.548093 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.548049    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fb7551-sj26p\" (UID: \"68271017-34eb-41ed-a4c9-d9d3db251d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"
Oct 31 04:52:16.548313 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.548112    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fb7551-sj26p\" (UID: \"68271017-34eb-41ed-a4c9-d9d3db251d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"
Oct 31 04:52:16.548313 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.548161    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbxhs\" (UniqueName: \"kubernetes.io/projected/68271017-34eb-41ed-a4c9-d9d3db251d7f-kube-api-access-zbxhs\") pod \"maas-api-revoke-sub-f0fb7551-sj26p\" (UID: \"68271017-34eb-41ed-a4c9-d9d3db251d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"
Oct 31 04:52:16.548771 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.548750    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fb7551-sj26p\" (UID: \"68271017-34eb-41ed-a4c9-d9d3db251d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"
Oct 31 04:52:16.550572 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.550552    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fb7551-sj26p\" (UID: \"68271017-34eb-41ed-a4c9-d9d3db251d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"
Oct 31 04:52:16.602714 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.602680    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbxhs\" (UniqueName: \"kubernetes.io/projected/68271017-34eb-41ed-a4c9-d9d3db251d7f-kube-api-access-zbxhs\") pod \"maas-api-revoke-sub-f0fb7551-sj26p\" (UID: \"68271017-34eb-41ed-a4c9-d9d3db251d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"
Oct 31 04:52:16.657733 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:16.657686    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"
Oct 31 04:52:17.021977 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:17.021899    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"]
Oct 31 04:52:17.025077 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:52:17.025037    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68271017_34eb_41ed_a4c9_d9d3db251d7f.slice/crio-49eae1ed304b33ffd0526eff649e8a8002d582aa2d3a521ab6c1dac91f0624d1 WatchSource:0}: Error finding container 49eae1ed304b33ffd0526eff649e8a8002d582aa2d3a521ab6c1dac91f0624d1: Status 404 returned error can't find the container with id 49eae1ed304b33ffd0526eff649e8a8002d582aa2d3a521ab6c1dac91f0624d1
Oct 31 04:52:17.183342 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:17.183306    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p" event={"ID":"68271017-34eb-41ed-a4c9-d9d3db251d7f","Type":"ContainerStarted","Data":"49eae1ed304b33ffd0526eff649e8a8002d582aa2d3a521ab6c1dac91f0624d1"}
Oct 31 04:52:18.190212 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:18.190180    5415 generic.go:356] "Generic (PLEG): container finished" podID="68271017-34eb-41ed-a4c9-d9d3db251d7f" containerID="930f50174697502360e9bd147744d33e495d6f63e05780d5f891c5e1014d057a" exitCode=0
Oct 31 04:52:18.190626 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:18.190271    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p" event={"ID":"68271017-34eb-41ed-a4c9-d9d3db251d7f","Type":"ContainerDied","Data":"930f50174697502360e9bd147744d33e495d6f63e05780d5f891c5e1014d057a"}
Oct 31 04:52:19.331667 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:19.331640    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"
Oct 31 04:52:19.378046 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:19.378009    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-xks-ca\") pod \"68271017-34eb-41ed-a4c9-d9d3db251d7f\" (UID: \"68271017-34eb-41ed-a4c9-d9d3db251d7f\") "
Oct 31 04:52:19.378257 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:19.378102    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68271017-34eb-41ed-a4c9-d9d3db251d7f-kube-api-access-zbxhs\" (UniqueName: \"kubernetes.io/projected/68271017-34eb-41ed-a4c9-d9d3db251d7f-kube-api-access-zbxhs\") pod \"68271017-34eb-41ed-a4c9-d9d3db251d7f\" (UID: \"68271017-34eb-41ed-a4c9-d9d3db251d7f\") "
Oct 31 04:52:19.378257 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:19.378132    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-openshift-ca\") pod \"68271017-34eb-41ed-a4c9-d9d3db251d7f\" (UID: \"68271017-34eb-41ed-a4c9-d9d3db251d7f\") "
Oct 31 04:52:19.378554 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:19.378531    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-openshift-ca" pod "68271017-34eb-41ed-a4c9-d9d3db251d7f" (UID: "68271017-34eb-41ed-a4c9-d9d3db251d7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:52:19.380296 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:19.380268    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-xks-ca" pod "68271017-34eb-41ed-a4c9-d9d3db251d7f" (UID: "68271017-34eb-41ed-a4c9-d9d3db251d7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:52:19.380296 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:19.380281    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68271017-34eb-41ed-a4c9-d9d3db251d7f-kube-api-access-zbxhs" pod "68271017-34eb-41ed-a4c9-d9d3db251d7f" (UID: "68271017-34eb-41ed-a4c9-d9d3db251d7f"). InnerVolumeSpecName "kube-api-access-zbxhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:52:19.478959 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:19.478854    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:52:19.478959 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:19.478898    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbxhs\" (UniqueName: \"kubernetes.io/projected/68271017-34eb-41ed-a4c9-d9d3db251d7f-kube-api-access-zbxhs\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:52:19.478959 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:19.478908    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68271017-34eb-41ed-a4c9-d9d3db251d7f-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:52:20.200180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:20.200139    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p" event={"ID":"68271017-34eb-41ed-a4c9-d9d3db251d7f","Type":"ContainerDied","Data":"49eae1ed304b33ffd0526eff649e8a8002d582aa2d3a521ab6c1dac91f0624d1"}
Oct 31 04:52:20.200180 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:20.200174    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"
Oct 31 04:52:20.200393 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:20.200201    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49eae1ed304b33ffd0526eff649e8a8002d582aa2d3a521ab6c1dac91f0624d1"
Oct 31 04:52:27.101971 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.101935    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"]
Oct 31 04:52:27.102456 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.102363    5415 cpu_manager.go:414] "containerMap: removing container" podUID="68271017-34eb-41ed-a4c9-d9d3db251d7f" containerName="revoke-keys"
Oct 31 04:52:27.102456 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.102375    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68271017-34eb-41ed-a4c9-d9d3db251d7f" containerName="revoke-keys"
Oct 31 04:52:27.102530 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.102460    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="68271017-34eb-41ed-a4c9-d9d3db251d7f" containerName="revoke-keys"
Oct 31 04:52:27.105790 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.105767    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"
Oct 31 04:52:27.155086 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.155048    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:52:27.202177 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.202145    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:52:27.203593 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.203565    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"]
Oct 31 04:52:27.256850 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.256814    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92nqk\" (UniqueName: \"kubernetes.io/projected/9d547054-9913-4eab-bb46-5d707acf471c-kube-api-access-92nqk\") pod \"maas-api-revoke-sub-8affa336-gkdkp\" (UID: \"9d547054-9913-4eab-bb46-5d707acf471c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"
Oct 31 04:52:27.256850 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.256854    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8affa336-gkdkp\" (UID: \"9d547054-9913-4eab-bb46-5d707acf471c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"
Oct 31 04:52:27.257099 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.256912    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8affa336-gkdkp\" (UID: \"9d547054-9913-4eab-bb46-5d707acf471c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"
Oct 31 04:52:27.358572 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.358485    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8affa336-gkdkp\" (UID: \"9d547054-9913-4eab-bb46-5d707acf471c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"
Oct 31 04:52:27.358729 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.358589    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92nqk\" (UniqueName: \"kubernetes.io/projected/9d547054-9913-4eab-bb46-5d707acf471c-kube-api-access-92nqk\") pod \"maas-api-revoke-sub-8affa336-gkdkp\" (UID: \"9d547054-9913-4eab-bb46-5d707acf471c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"
Oct 31 04:52:27.358729 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.358625    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8affa336-gkdkp\" (UID: \"9d547054-9913-4eab-bb46-5d707acf471c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"
Oct 31 04:52:27.359239 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.359217    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8affa336-gkdkp\" (UID: \"9d547054-9913-4eab-bb46-5d707acf471c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"
Oct 31 04:52:27.360992 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.360976    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8affa336-gkdkp\" (UID: \"9d547054-9913-4eab-bb46-5d707acf471c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"
Oct 31 04:52:27.412735 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.412700    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92nqk\" (UniqueName: \"kubernetes.io/projected/9d547054-9913-4eab-bb46-5d707acf471c-kube-api-access-92nqk\") pod \"maas-api-revoke-sub-8affa336-gkdkp\" (UID: \"9d547054-9913-4eab-bb46-5d707acf471c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"
Oct 31 04:52:27.419543 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.419519    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"
Oct 31 04:52:27.782629 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:27.782604    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"]
Oct 31 04:52:27.785432 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:52:27.785400    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d547054_9913_4eab_bb46_5d707acf471c.slice/crio-77e9d68d8c76cc8bde79d5b96641a54605c1edc238cb9956e4a4b860723a729d WatchSource:0}: Error finding container 77e9d68d8c76cc8bde79d5b96641a54605c1edc238cb9956e4a4b860723a729d: Status 404 returned error can't find the container with id 77e9d68d8c76cc8bde79d5b96641a54605c1edc238cb9956e4a4b860723a729d
Oct 31 04:52:28.239250 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:28.239210    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp" event={"ID":"9d547054-9913-4eab-bb46-5d707acf471c","Type":"ContainerStarted","Data":"ab4c9ce7dfdcdc5dfe8ec49d93b510473c73bada5310aa5371244d5137b06a09"}
Oct 31 04:52:28.239654 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:28.239268    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp" event={"ID":"9d547054-9913-4eab-bb46-5d707acf471c","Type":"ContainerStarted","Data":"77e9d68d8c76cc8bde79d5b96641a54605c1edc238cb9956e4a4b860723a729d"}
Oct 31 04:52:28.348144 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:28.348092    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp" podStartSLOduration=1.348077048 podStartE2EDuration="1.348077048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:52:28.347108837 +0000 UTC m=+6289.907588573" watchObservedRunningTime="2026-10-31 04:52:28.348077048 +0000 UTC m=+6289.908556814"
Oct 31 04:52:29.246441 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:29.246406    5415 generic.go:356] "Generic (PLEG): container finished" podID="9d547054-9913-4eab-bb46-5d707acf471c" containerID="ab4c9ce7dfdcdc5dfe8ec49d93b510473c73bada5310aa5371244d5137b06a09" exitCode=0
Oct 31 04:52:29.246910 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:29.246492    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp" event={"ID":"9d547054-9913-4eab-bb46-5d707acf471c","Type":"ContainerDied","Data":"ab4c9ce7dfdcdc5dfe8ec49d93b510473c73bada5310aa5371244d5137b06a09"}
Oct 31 04:52:30.387576 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:30.387548    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"
Oct 31 04:52:30.488796 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:30.488756    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-openshift-ca\") pod \"9d547054-9913-4eab-bb46-5d707acf471c\" (UID: \"9d547054-9913-4eab-bb46-5d707acf471c\") "
Oct 31 04:52:30.489026 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:30.488858    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-xks-ca\") pod \"9d547054-9913-4eab-bb46-5d707acf471c\" (UID: \"9d547054-9913-4eab-bb46-5d707acf471c\") "
Oct 31 04:52:30.489026 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:30.488919    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d547054-9913-4eab-bb46-5d707acf471c-kube-api-access-92nqk\" (UniqueName: \"kubernetes.io/projected/9d547054-9913-4eab-bb46-5d707acf471c-kube-api-access-92nqk\") pod \"9d547054-9913-4eab-bb46-5d707acf471c\" (UID: \"9d547054-9913-4eab-bb46-5d707acf471c\") "
Oct 31 04:52:30.489291 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:30.489259    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-openshift-ca" pod "9d547054-9913-4eab-bb46-5d707acf471c" (UID: "9d547054-9913-4eab-bb46-5d707acf471c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:52:30.491091 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:30.491067    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-xks-ca" pod "9d547054-9913-4eab-bb46-5d707acf471c" (UID: "9d547054-9913-4eab-bb46-5d707acf471c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:52:30.491262 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:30.491238    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d547054-9913-4eab-bb46-5d707acf471c-kube-api-access-92nqk" pod "9d547054-9913-4eab-bb46-5d707acf471c" (UID: "9d547054-9913-4eab-bb46-5d707acf471c"). InnerVolumeSpecName "kube-api-access-92nqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:52:30.590127 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:30.590083    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:52:30.590127 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:30.590118    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d547054-9913-4eab-bb46-5d707acf471c-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:52:30.590127 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:30.590128    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92nqk\" (UniqueName: \"kubernetes.io/projected/9d547054-9913-4eab-bb46-5d707acf471c-kube-api-access-92nqk\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:52:31.258264 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:31.258227    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp" event={"ID":"9d547054-9913-4eab-bb46-5d707acf471c","Type":"ContainerDied","Data":"77e9d68d8c76cc8bde79d5b96641a54605c1edc238cb9956e4a4b860723a729d"}
Oct 31 04:52:31.258449 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:31.258282    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77e9d68d8c76cc8bde79d5b96641a54605c1edc238cb9956e4a4b860723a729d"
Oct 31 04:52:31.258449 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:52:31.258244    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"
Oct 31 04:52:46.175239 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:52:46.175155    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:53:03.824968 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:03.824927    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"]
Oct 31 04:53:03.826027 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:03.825993    5415 cpu_manager.go:414] "containerMap: removing container" podUID="9d547054-9913-4eab-bb46-5d707acf471c" containerName="revoke-keys"
Oct 31 04:53:03.826027 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:03.826026    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d547054-9913-4eab-bb46-5d707acf471c" containerName="revoke-keys"
Oct 31 04:53:03.826284 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:03.826270    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d547054-9913-4eab-bb46-5d707acf471c" containerName="revoke-keys"
Oct 31 04:53:03.835485 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:03.835449    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"
Oct 31 04:53:03.884861 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:03.884824    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"]
Oct 31 04:53:03.924548 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:03.924504    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq6kv\" (UniqueName: \"kubernetes.io/projected/e14ab89e-f2d6-4615-84bd-1a380d2440c9-kube-api-access-tq6kv\") pod \"maas-api-revoke-sub-b907400c-8lwps\" (UID: \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"
Oct 31 04:53:03.924775 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:03.924571    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b907400c-8lwps\" (UID: \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"
Oct 31 04:53:03.924775 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:03.924690    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b907400c-8lwps\" (UID: \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"
Oct 31 04:53:03.930749 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:03.930721    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:53:03.930906 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:03.930774    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:53:04.025849 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:04.025805    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b907400c-8lwps\" (UID: \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"
Oct 31 04:53:04.026061 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:04.025931    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq6kv\" (UniqueName: \"kubernetes.io/projected/e14ab89e-f2d6-4615-84bd-1a380d2440c9-kube-api-access-tq6kv\") pod \"maas-api-revoke-sub-b907400c-8lwps\" (UID: \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"
Oct 31 04:53:04.026061 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:04.025989    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b907400c-8lwps\" (UID: \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"
Oct 31 04:53:04.026685 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:04.026666    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b907400c-8lwps\" (UID: \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"
Oct 31 04:53:04.028485 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:04.028460    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b907400c-8lwps\" (UID: \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"
Oct 31 04:53:04.080313 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:04.080223    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq6kv\" (UniqueName: \"kubernetes.io/projected/e14ab89e-f2d6-4615-84bd-1a380d2440c9-kube-api-access-tq6kv\") pod \"maas-api-revoke-sub-b907400c-8lwps\" (UID: \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"
Oct 31 04:53:04.149365 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:04.149324    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"
Oct 31 04:53:04.515147 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:04.515125    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"]
Oct 31 04:53:04.517909 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:53:04.517854    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14ab89e_f2d6_4615_84bd_1a380d2440c9.slice/crio-2af3be7d8587e807e8031472ddbfe5d88cb7e49bda517ccaaeca42e31ed10790 WatchSource:0}: Error finding container 2af3be7d8587e807e8031472ddbfe5d88cb7e49bda517ccaaeca42e31ed10790: Status 404 returned error can't find the container with id 2af3be7d8587e807e8031472ddbfe5d88cb7e49bda517ccaaeca42e31ed10790
Oct 31 04:53:05.484342 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:05.484307    5415 generic.go:356] "Generic (PLEG): container finished" podID="e14ab89e-f2d6-4615-84bd-1a380d2440c9" containerID="cd3774f98ec0ffdbc9df37ed6802e70e9e3b35807ed26965d9e7eca2ca37177a" exitCode=0
Oct 31 04:53:05.484775 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:05.484395    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps" event={"ID":"e14ab89e-f2d6-4615-84bd-1a380d2440c9","Type":"ContainerDied","Data":"cd3774f98ec0ffdbc9df37ed6802e70e9e3b35807ed26965d9e7eca2ca37177a"}
Oct 31 04:53:05.484775 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:05.484458    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps" event={"ID":"e14ab89e-f2d6-4615-84bd-1a380d2440c9","Type":"ContainerStarted","Data":"2af3be7d8587e807e8031472ddbfe5d88cb7e49bda517ccaaeca42e31ed10790"}
Oct 31 04:53:06.620973 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:06.620943    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"
Oct 31 04:53:06.751829 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:06.751745    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-xks-ca\") pod \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\" (UID: \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\") "
Oct 31 04:53:06.752000 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:06.751872    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14ab89e-f2d6-4615-84bd-1a380d2440c9-kube-api-access-tq6kv\" (UniqueName: \"kubernetes.io/projected/e14ab89e-f2d6-4615-84bd-1a380d2440c9-kube-api-access-tq6kv\") pod \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\" (UID: \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\") "
Oct 31 04:53:06.752000 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:06.751919    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-openshift-ca\") pod \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\" (UID: \"e14ab89e-f2d6-4615-84bd-1a380d2440c9\") "
Oct 31 04:53:06.752340 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:06.752311    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-openshift-ca" pod "e14ab89e-f2d6-4615-84bd-1a380d2440c9" (UID: "e14ab89e-f2d6-4615-84bd-1a380d2440c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:53:06.754025 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:06.753997    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-xks-ca" pod "e14ab89e-f2d6-4615-84bd-1a380d2440c9" (UID: "e14ab89e-f2d6-4615-84bd-1a380d2440c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:53:06.754133 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:06.754020    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14ab89e-f2d6-4615-84bd-1a380d2440c9-kube-api-access-tq6kv" pod "e14ab89e-f2d6-4615-84bd-1a380d2440c9" (UID: "e14ab89e-f2d6-4615-84bd-1a380d2440c9"). InnerVolumeSpecName "kube-api-access-tq6kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:53:06.852988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:06.852947    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq6kv\" (UniqueName: \"kubernetes.io/projected/e14ab89e-f2d6-4615-84bd-1a380d2440c9-kube-api-access-tq6kv\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:53:06.852988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:06.852980    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:53:06.852988 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:06.852993    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14ab89e-f2d6-4615-84bd-1a380d2440c9-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:53:07.494604 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:07.494572    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"
Oct 31 04:53:07.494604 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:07.494594    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps" event={"ID":"e14ab89e-f2d6-4615-84bd-1a380d2440c9","Type":"ContainerDied","Data":"2af3be7d8587e807e8031472ddbfe5d88cb7e49bda517ccaaeca42e31ed10790"}
Oct 31 04:53:07.494833 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:07.494652    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2af3be7d8587e807e8031472ddbfe5d88cb7e49bda517ccaaeca42e31ed10790"
Oct 31 04:53:16.175943 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:53:16.175819    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:53:45.402078 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.402037    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"]
Oct 31 04:53:45.402487 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.402451    5415 cpu_manager.go:414] "containerMap: removing container" podUID="e14ab89e-f2d6-4615-84bd-1a380d2440c9" containerName="revoke-keys"
Oct 31 04:53:45.402487 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.402463    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14ab89e-f2d6-4615-84bd-1a380d2440c9" containerName="revoke-keys"
Oct 31 04:53:45.402565 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.402549    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14ab89e-f2d6-4615-84bd-1a380d2440c9" containerName="revoke-keys"
Oct 31 04:53:45.406045 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.406023    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"
Oct 31 04:53:45.502753 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.502721    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:53:45.502753 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.502724    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:53:45.504488 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.504469    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"]
Oct 31 04:53:45.545501 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.545463    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04dc118f-6ds4t\" (UID: \"1d5a92ba-20bc-40e6-986a-d0449d611210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"
Oct 31 04:53:45.545693 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.545510    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64sdd\" (UniqueName: \"kubernetes.io/projected/1d5a92ba-20bc-40e6-986a-d0449d611210-kube-api-access-64sdd\") pod \"maas-api-revoke-sub-04dc118f-6ds4t\" (UID: \"1d5a92ba-20bc-40e6-986a-d0449d611210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"
Oct 31 04:53:45.545693 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.545596    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04dc118f-6ds4t\" (UID: \"1d5a92ba-20bc-40e6-986a-d0449d611210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"
Oct 31 04:53:45.647066 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.647031    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64sdd\" (UniqueName: \"kubernetes.io/projected/1d5a92ba-20bc-40e6-986a-d0449d611210-kube-api-access-64sdd\") pod \"maas-api-revoke-sub-04dc118f-6ds4t\" (UID: \"1d5a92ba-20bc-40e6-986a-d0449d611210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"
Oct 31 04:53:45.647262 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.647087    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04dc118f-6ds4t\" (UID: \"1d5a92ba-20bc-40e6-986a-d0449d611210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"
Oct 31 04:53:45.647262 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.647209    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04dc118f-6ds4t\" (UID: \"1d5a92ba-20bc-40e6-986a-d0449d611210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"
Oct 31 04:53:45.647741 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.647716    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04dc118f-6ds4t\" (UID: \"1d5a92ba-20bc-40e6-986a-d0449d611210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"
Oct 31 04:53:45.649427 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.649406    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04dc118f-6ds4t\" (UID: \"1d5a92ba-20bc-40e6-986a-d0449d611210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"
Oct 31 04:53:45.701347 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.701271    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64sdd\" (UniqueName: \"kubernetes.io/projected/1d5a92ba-20bc-40e6-986a-d0449d611210-kube-api-access-64sdd\") pod \"maas-api-revoke-sub-04dc118f-6ds4t\" (UID: \"1d5a92ba-20bc-40e6-986a-d0449d611210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"
Oct 31 04:53:45.718577 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:45.718543    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"
Oct 31 04:53:46.083300 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:46.083277    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"]
Oct 31 04:53:46.086223 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:53:46.086194    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5a92ba_20bc_40e6_986a_d0449d611210.slice/crio-150916b80f043610227a8d59a68e2bd65c1a52fd1834e7fc15cc4bd744eb9837 WatchSource:0}: Error finding container 150916b80f043610227a8d59a68e2bd65c1a52fd1834e7fc15cc4bd744eb9837: Status 404 returned error can't find the container with id 150916b80f043610227a8d59a68e2bd65c1a52fd1834e7fc15cc4bd744eb9837
Oct 31 04:53:46.174075 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:53:46.174035    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:53:46.746913 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:46.746797    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t" event={"ID":"1d5a92ba-20bc-40e6-986a-d0449d611210","Type":"ContainerStarted","Data":"cb61046fe45c84f659a8277224e42bc561beb210639464836f86e364ef93ef5f"}
Oct 31 04:53:46.746913 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:46.746850    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t" event={"ID":"1d5a92ba-20bc-40e6-986a-d0449d611210","Type":"ContainerStarted","Data":"150916b80f043610227a8d59a68e2bd65c1a52fd1834e7fc15cc4bd744eb9837"}
Oct 31 04:53:46.855361 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:46.855295    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t" podStartSLOduration=1.855278324 podStartE2EDuration="1.855278324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:53:46.854116273 +0000 UTC m=+6368.414596027" watchObservedRunningTime="2026-10-31 04:53:46.855278324 +0000 UTC m=+6368.415758055"
Oct 31 04:53:47.753466 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:47.753423    5415 generic.go:356] "Generic (PLEG): container finished" podID="1d5a92ba-20bc-40e6-986a-d0449d611210" containerID="cb61046fe45c84f659a8277224e42bc561beb210639464836f86e364ef93ef5f" exitCode=0
Oct 31 04:53:47.754144 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:47.753509    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t" event={"ID":"1d5a92ba-20bc-40e6-986a-d0449d611210","Type":"ContainerDied","Data":"cb61046fe45c84f659a8277224e42bc561beb210639464836f86e364ef93ef5f"}
Oct 31 04:53:48.893758 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:48.893728    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"
Oct 31 04:53:48.979476 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:48.979442    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-openshift-ca\") pod \"1d5a92ba-20bc-40e6-986a-d0449d611210\" (UID: \"1d5a92ba-20bc-40e6-986a-d0449d611210\") "
Oct 31 04:53:48.979659 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:48.979549    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-xks-ca\") pod \"1d5a92ba-20bc-40e6-986a-d0449d611210\" (UID: \"1d5a92ba-20bc-40e6-986a-d0449d611210\") "
Oct 31 04:53:48.979659 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:48.979607    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d5a92ba-20bc-40e6-986a-d0449d611210-kube-api-access-64sdd\" (UniqueName: \"kubernetes.io/projected/1d5a92ba-20bc-40e6-986a-d0449d611210-kube-api-access-64sdd\") pod \"1d5a92ba-20bc-40e6-986a-d0449d611210\" (UID: \"1d5a92ba-20bc-40e6-986a-d0449d611210\") "
Oct 31 04:53:48.979919 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:48.979861    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-openshift-ca" pod "1d5a92ba-20bc-40e6-986a-d0449d611210" (UID: "1d5a92ba-20bc-40e6-986a-d0449d611210"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:53:48.981624 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:48.981597    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-xks-ca" pod "1d5a92ba-20bc-40e6-986a-d0449d611210" (UID: "1d5a92ba-20bc-40e6-986a-d0449d611210"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:53:48.981819 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:48.981800    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5a92ba-20bc-40e6-986a-d0449d611210-kube-api-access-64sdd" pod "1d5a92ba-20bc-40e6-986a-d0449d611210" (UID: "1d5a92ba-20bc-40e6-986a-d0449d611210"). InnerVolumeSpecName "kube-api-access-64sdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:53:49.080770 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:49.080687    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:53:49.080770 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:49.080717    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64sdd\" (UniqueName: \"kubernetes.io/projected/1d5a92ba-20bc-40e6-986a-d0449d611210-kube-api-access-64sdd\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:53:49.080770 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:49.080728    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5a92ba-20bc-40e6-986a-d0449d611210-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:53:49.764078 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:49.764042    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"
Oct 31 04:53:49.764078 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:49.764069    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t" event={"ID":"1d5a92ba-20bc-40e6-986a-d0449d611210","Type":"ContainerDied","Data":"150916b80f043610227a8d59a68e2bd65c1a52fd1834e7fc15cc4bd744eb9837"}
Oct 31 04:53:49.764315 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:49.764120    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="150916b80f043610227a8d59a68e2bd65c1a52fd1834e7fc15cc4bd744eb9837"
Oct 31 04:53:56.146523 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.146489    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"]
Oct 31 04:53:56.146952 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.146908    5415 cpu_manager.go:414] "containerMap: removing container" podUID="1d5a92ba-20bc-40e6-986a-d0449d611210" containerName="revoke-keys"
Oct 31 04:53:56.146952 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.146920    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d5a92ba-20bc-40e6-986a-d0449d611210" containerName="revoke-keys"
Oct 31 04:53:56.147064 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.147005    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d5a92ba-20bc-40e6-986a-d0449d611210" containerName="revoke-keys"
Oct 31 04:53:56.150489 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.150466    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"
Oct 31 04:53:56.200488 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.200449    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:53:56.247371 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.247336    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:53:56.249008 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.248985    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"]
Oct 31 04:53:56.251834 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.251813    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c89297c-ztfr7\" (UID: \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"
Oct 31 04:53:56.251918 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.251849    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwn7\" (UniqueName: \"kubernetes.io/projected/70d3e47e-18e6-4255-8fa4-824e7ed524d1-kube-api-access-smwn7\") pod \"maas-api-revoke-sub-1c89297c-ztfr7\" (UID: \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"
Oct 31 04:53:56.251991 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.251968    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c89297c-ztfr7\" (UID: \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"
Oct 31 04:53:56.353449 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.353406    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c89297c-ztfr7\" (UID: \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"
Oct 31 04:53:56.353642 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.353469    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smwn7\" (UniqueName: \"kubernetes.io/projected/70d3e47e-18e6-4255-8fa4-824e7ed524d1-kube-api-access-smwn7\") pod \"maas-api-revoke-sub-1c89297c-ztfr7\" (UID: \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"
Oct 31 04:53:56.353642 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.353541    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c89297c-ztfr7\" (UID: \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"
Oct 31 04:53:56.354120 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.354095    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c89297c-ztfr7\" (UID: \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"
Oct 31 04:53:56.356010 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.355988    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c89297c-ztfr7\" (UID: \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"
Oct 31 04:53:56.408168 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.408086    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwn7\" (UniqueName: \"kubernetes.io/projected/70d3e47e-18e6-4255-8fa4-824e7ed524d1-kube-api-access-smwn7\") pod \"maas-api-revoke-sub-1c89297c-ztfr7\" (UID: \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"
Oct 31 04:53:56.464520 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.464485    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"
Oct 31 04:53:56.832799 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:56.832755    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"]
Oct 31 04:53:56.836285 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:53:56.836252    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d3e47e_18e6_4255_8fa4_824e7ed524d1.slice/crio-30cb743f2562bb4b06f75594465519965df209ba1a6d5c329d0be8a20f2be4dc WatchSource:0}: Error finding container 30cb743f2562bb4b06f75594465519965df209ba1a6d5c329d0be8a20f2be4dc: Status 404 returned error can't find the container with id 30cb743f2562bb4b06f75594465519965df209ba1a6d5c329d0be8a20f2be4dc
Oct 31 04:53:57.802605 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:57.802572    5415 generic.go:356] "Generic (PLEG): container finished" podID="70d3e47e-18e6-4255-8fa4-824e7ed524d1" containerID="1664db4fd080e43797baaaf7f54b68a4864b1ebf71fdfbbb0cccdac66b8ecdd8" exitCode=0
Oct 31 04:53:57.802996 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:57.802660    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7" event={"ID":"70d3e47e-18e6-4255-8fa4-824e7ed524d1","Type":"ContainerDied","Data":"1664db4fd080e43797baaaf7f54b68a4864b1ebf71fdfbbb0cccdac66b8ecdd8"}
Oct 31 04:53:57.802996 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:57.802717    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7" event={"ID":"70d3e47e-18e6-4255-8fa4-824e7ed524d1","Type":"ContainerStarted","Data":"30cb743f2562bb4b06f75594465519965df209ba1a6d5c329d0be8a20f2be4dc"}
Oct 31 04:53:58.945486 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:58.945457    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"
Oct 31 04:53:59.089991 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:59.089957    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-xks-ca\") pod \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\" (UID: \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\") "
Oct 31 04:53:59.090162 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:59.090022    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-openshift-ca\") pod \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\" (UID: \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\") "
Oct 31 04:53:59.090162 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:59.090128    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d3e47e-18e6-4255-8fa4-824e7ed524d1-kube-api-access-smwn7\" (UniqueName: \"kubernetes.io/projected/70d3e47e-18e6-4255-8fa4-824e7ed524d1-kube-api-access-smwn7\") pod \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\" (UID: \"70d3e47e-18e6-4255-8fa4-824e7ed524d1\") "
Oct 31 04:53:59.090450 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:59.090422    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-openshift-ca" pod "70d3e47e-18e6-4255-8fa4-824e7ed524d1" (UID: "70d3e47e-18e6-4255-8fa4-824e7ed524d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:53:59.092090 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:59.092069    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-xks-ca" pod "70d3e47e-18e6-4255-8fa4-824e7ed524d1" (UID: "70d3e47e-18e6-4255-8fa4-824e7ed524d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:53:59.092155 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:59.092138    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d3e47e-18e6-4255-8fa4-824e7ed524d1-kube-api-access-smwn7" pod "70d3e47e-18e6-4255-8fa4-824e7ed524d1" (UID: "70d3e47e-18e6-4255-8fa4-824e7ed524d1"). InnerVolumeSpecName "kube-api-access-smwn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:53:59.191534 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:59.191482    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smwn7\" (UniqueName: \"kubernetes.io/projected/70d3e47e-18e6-4255-8fa4-824e7ed524d1-kube-api-access-smwn7\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:53:59.191534 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:59.191527    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:53:59.191534 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:59.191541    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d3e47e-18e6-4255-8fa4-824e7ed524d1-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:53:59.813369 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:59.813331    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7" event={"ID":"70d3e47e-18e6-4255-8fa4-824e7ed524d1","Type":"ContainerDied","Data":"30cb743f2562bb4b06f75594465519965df209ba1a6d5c329d0be8a20f2be4dc"}
Oct 31 04:53:59.813369 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:59.813363    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"
Oct 31 04:53:59.813656 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:53:59.813383    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30cb743f2562bb4b06f75594465519965df209ba1a6d5c329d0be8a20f2be4dc"
Oct 31 04:54:16.180345 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:54:16.180138    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:54:18.094902 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:18.088026    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wlt76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:54:18.094902 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:18.088169    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76" podUID="c0469dfd-14dc-458a-9b3e-d5b0ecc25000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:54:44.096062 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.096021    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"]
Oct 31 04:54:44.096616 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.096596    5415 cpu_manager.go:414] "containerMap: removing container" podUID="70d3e47e-18e6-4255-8fa4-824e7ed524d1" containerName="revoke-keys"
Oct 31 04:54:44.096698 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.096620    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d3e47e-18e6-4255-8fa4-824e7ed524d1" containerName="revoke-keys"
Oct 31 04:54:44.096775 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.096761    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d3e47e-18e6-4255-8fa4-824e7ed524d1" containerName="revoke-keys"
Oct 31 04:54:44.100338 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.100316    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"
Oct 31 04:54:44.150044 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.150007    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:54:44.197190 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.197153    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:54:44.198687 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.198662    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"]
Oct 31 04:54:44.225096 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.225054    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7g69\" (UniqueName: \"kubernetes.io/projected/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-kube-api-access-r7g69\") pod \"maas-api-revoke-sub-4d8262f3-v4tvc\" (UID: \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"
Oct 31 04:54:44.225300 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.225131    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8262f3-v4tvc\" (UID: \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"
Oct 31 04:54:44.225300 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.225234    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8262f3-v4tvc\" (UID: \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"
Oct 31 04:54:44.326643 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.326605    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7g69\" (UniqueName: \"kubernetes.io/projected/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-kube-api-access-r7g69\") pod \"maas-api-revoke-sub-4d8262f3-v4tvc\" (UID: \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"
Oct 31 04:54:44.326866 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.326684    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8262f3-v4tvc\" (UID: \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"
Oct 31 04:54:44.326866 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.326747    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8262f3-v4tvc\" (UID: \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"
Oct 31 04:54:44.327450 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.327422    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8262f3-v4tvc\" (UID: \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"
Oct 31 04:54:44.329188 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.329170    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8262f3-v4tvc\" (UID: \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"
Oct 31 04:54:44.380911 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.380810    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7g69\" (UniqueName: \"kubernetes.io/projected/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-kube-api-access-r7g69\") pod \"maas-api-revoke-sub-4d8262f3-v4tvc\" (UID: \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"
Oct 31 04:54:44.413600 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.413559    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"
Oct 31 04:54:44.778943 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:44.778919    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"]
Oct 31 04:54:44.781278 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:54:44.781233    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5611f194_dbb2_47bd_b913_3ffc8ac4ea92.slice/crio-6cbe11481051209dcaea7475f1396ee24013d57480682189d2bef532989cb8e3 WatchSource:0}: Error finding container 6cbe11481051209dcaea7475f1396ee24013d57480682189d2bef532989cb8e3: Status 404 returned error can't find the container with id 6cbe11481051209dcaea7475f1396ee24013d57480682189d2bef532989cb8e3
Oct 31 04:54:45.093775 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:45.093734    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc" event={"ID":"5611f194-dbb2-47bd-b913-3ffc8ac4ea92","Type":"ContainerStarted","Data":"73d01509469407ea685ce788ce443c7efefa7c5f7fa351f579ed396cab8280ef"}
Oct 31 04:54:45.093977 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:45.093804    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc" event={"ID":"5611f194-dbb2-47bd-b913-3ffc8ac4ea92","Type":"ContainerStarted","Data":"6cbe11481051209dcaea7475f1396ee24013d57480682189d2bef532989cb8e3"}
Oct 31 04:54:45.203098 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:45.203040    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc" podStartSLOduration=1.203023079 podStartE2EDuration="1.203023079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:54:45.20140263 +0000 UTC m=+6426.761882360" watchObservedRunningTime="2026-10-31 04:54:45.203023079 +0000 UTC m=+6426.763502881"
Oct 31 04:54:46.107264 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:46.107227    5415 generic.go:356] "Generic (PLEG): container finished" podID="5611f194-dbb2-47bd-b913-3ffc8ac4ea92" containerID="73d01509469407ea685ce788ce443c7efefa7c5f7fa351f579ed396cab8280ef" exitCode=0
Oct 31 04:54:46.107461 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:46.107297    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc" event={"ID":"5611f194-dbb2-47bd-b913-3ffc8ac4ea92","Type":"ContainerDied","Data":"73d01509469407ea685ce788ce443c7efefa7c5f7fa351f579ed396cab8280ef"}
Oct 31 04:54:46.175093 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:54:46.175054    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:54:47.248644 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:47.248616    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"
Oct 31 04:54:47.357164 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:47.357129    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-kube-api-access-r7g69\" (UniqueName: \"kubernetes.io/projected/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-kube-api-access-r7g69\") pod \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\" (UID: \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\") "
Oct 31 04:54:47.357324 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:47.357192    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-xks-ca\") pod \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\" (UID: \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\") "
Oct 31 04:54:47.357324 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:47.357301    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-openshift-ca\") pod \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\" (UID: \"5611f194-dbb2-47bd-b913-3ffc8ac4ea92\") "
Oct 31 04:54:47.357745 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:47.357717    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-openshift-ca" pod "5611f194-dbb2-47bd-b913-3ffc8ac4ea92" (UID: "5611f194-dbb2-47bd-b913-3ffc8ac4ea92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:54:47.359567 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:47.359541    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-xks-ca" pod "5611f194-dbb2-47bd-b913-3ffc8ac4ea92" (UID: "5611f194-dbb2-47bd-b913-3ffc8ac4ea92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:54:47.359703 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:47.359574    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-kube-api-access-r7g69" pod "5611f194-dbb2-47bd-b913-3ffc8ac4ea92" (UID: "5611f194-dbb2-47bd-b913-3ffc8ac4ea92"). InnerVolumeSpecName "kube-api-access-r7g69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:54:47.458810 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:47.458776    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7g69\" (UniqueName: \"kubernetes.io/projected/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-kube-api-access-r7g69\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:54:47.458810 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:47.458807    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:54:47.458810 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:47.458819    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5611f194-dbb2-47bd-b913-3ffc8ac4ea92-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:54:48.117457 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:48.117426    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc"
Oct 31 04:54:48.117680 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:48.117427    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8262f3-v4tvc" event={"ID":"5611f194-dbb2-47bd-b913-3ffc8ac4ea92","Type":"ContainerDied","Data":"6cbe11481051209dcaea7475f1396ee24013d57480682189d2bef532989cb8e3"}
Oct 31 04:54:48.117680 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:48.117556    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cbe11481051209dcaea7475f1396ee24013d57480682189d2bef532989cb8e3"
Oct 31 04:54:54.805718 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:54.805678    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"]
Oct 31 04:54:54.806230 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:54.806114    5415 cpu_manager.go:414] "containerMap: removing container" podUID="5611f194-dbb2-47bd-b913-3ffc8ac4ea92" containerName="revoke-keys"
Oct 31 04:54:54.806230 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:54.806127    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5611f194-dbb2-47bd-b913-3ffc8ac4ea92" containerName="revoke-keys"
Oct 31 04:54:54.806230 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:54.806193    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="5611f194-dbb2-47bd-b913-3ffc8ac4ea92" containerName="revoke-keys"
Oct 31 04:54:54.809655 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:54.809613    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"
Oct 31 04:54:54.859065 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:54.859032    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:54:54.906301 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:54.906227    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:54:54.907909 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:54.907864    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"]
Oct 31 04:54:54.927970 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:54.927939    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3fa8702-qff4w\" (UID: \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"
Oct 31 04:54:54.928134 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:54.927979    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3fa8702-qff4w\" (UID: \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"
Oct 31 04:54:54.928134 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:54.928007    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdbg2\" (UniqueName: \"kubernetes.io/projected/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-kube-api-access-qdbg2\") pod \"maas-api-revoke-sub-b3fa8702-qff4w\" (UID: \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"
Oct 31 04:54:55.029304 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:55.029258    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3fa8702-qff4w\" (UID: \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"
Oct 31 04:54:55.029506 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:55.029321    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3fa8702-qff4w\" (UID: \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"
Oct 31 04:54:55.029506 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:55.029347    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdbg2\" (UniqueName: \"kubernetes.io/projected/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-kube-api-access-qdbg2\") pod \"maas-api-revoke-sub-b3fa8702-qff4w\" (UID: \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"
Oct 31 04:54:55.030158 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:55.030132    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3fa8702-qff4w\" (UID: \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"
Oct 31 04:54:55.031765 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:55.031737    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3fa8702-qff4w\" (UID: \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"
Oct 31 04:54:55.083831 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:55.083806    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdbg2\" (UniqueName: \"kubernetes.io/projected/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-kube-api-access-qdbg2\") pod \"maas-api-revoke-sub-b3fa8702-qff4w\" (UID: \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"
Oct 31 04:54:55.124684 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:55.124646    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"
Oct 31 04:54:55.490176 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:55.490045    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"]
Oct 31 04:54:55.493287 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:54:55.493253    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ded1b2e_cb23_403a_bd07_3dcfaa0bea95.slice/crio-bca82d9baed57d8b7c7c622cda68f922d5bf47eb1e0fa1e11e5ae11ac1accd07 WatchSource:0}: Error finding container bca82d9baed57d8b7c7c622cda68f922d5bf47eb1e0fa1e11e5ae11ac1accd07: Status 404 returned error can't find the container with id bca82d9baed57d8b7c7c622cda68f922d5bf47eb1e0fa1e11e5ae11ac1accd07
Oct 31 04:54:56.157251 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:56.157216    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w" event={"ID":"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95","Type":"ContainerStarted","Data":"eaa01268e4f1efe9157830a2738ddc06644687fe87b71ab0af9c6b4c796d663c"}
Oct 31 04:54:56.157667 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:56.157274    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w" event={"ID":"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95","Type":"ContainerStarted","Data":"bca82d9baed57d8b7c7c622cda68f922d5bf47eb1e0fa1e11e5ae11ac1accd07"}
Oct 31 04:54:56.264633 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:56.264584    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w" podStartSLOduration=2.264567785 podStartE2EDuration="2.264567785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:54:56.264048691 +0000 UTC m=+6437.824528422" watchObservedRunningTime="2026-10-31 04:54:56.264567785 +0000 UTC m=+6437.825047512"
Oct 31 04:54:57.166724 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:57.166687    5415 generic.go:356] "Generic (PLEG): container finished" podID="9ded1b2e-cb23-403a-bd07-3dcfaa0bea95" containerID="eaa01268e4f1efe9157830a2738ddc06644687fe87b71ab0af9c6b4c796d663c" exitCode=0
Oct 31 04:54:57.167447 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:57.167286    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w" event={"ID":"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95","Type":"ContainerDied","Data":"eaa01268e4f1efe9157830a2738ddc06644687fe87b71ab0af9c6b4c796d663c"}
Oct 31 04:54:58.603486 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:58.603102    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"
Oct 31 04:54:58.629989 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:58.629951    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-kube-api-access-qdbg2\" (UniqueName: \"kubernetes.io/projected/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-kube-api-access-qdbg2\") pod \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\" (UID: \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\") "
Oct 31 04:54:58.631002 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:58.630976    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-xks-ca\") pod \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\" (UID: \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\") "
Oct 31 04:54:58.631155 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:58.631040    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-openshift-ca\") pod \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\" (UID: \"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95\") "
Oct 31 04:54:58.631698 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:58.631674    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-openshift-ca" pod "9ded1b2e-cb23-403a-bd07-3dcfaa0bea95" (UID: "9ded1b2e-cb23-403a-bd07-3dcfaa0bea95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:54:58.633270 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:58.633244    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-kube-api-access-qdbg2" pod "9ded1b2e-cb23-403a-bd07-3dcfaa0bea95" (UID: "9ded1b2e-cb23-403a-bd07-3dcfaa0bea95"). InnerVolumeSpecName "kube-api-access-qdbg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:54:58.633685 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:58.633669    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-xks-ca" pod "9ded1b2e-cb23-403a-bd07-3dcfaa0bea95" (UID: "9ded1b2e-cb23-403a-bd07-3dcfaa0bea95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:54:58.732131 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:58.732086    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:54:58.732131 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:58.732128    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:54:58.732417 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:58.732144    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdbg2\" (UniqueName: \"kubernetes.io/projected/9ded1b2e-cb23-403a-bd07-3dcfaa0bea95-kube-api-access-qdbg2\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:54:59.206703 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:59.206662    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w" event={"ID":"9ded1b2e-cb23-403a-bd07-3dcfaa0bea95","Type":"ContainerDied","Data":"bca82d9baed57d8b7c7c622cda68f922d5bf47eb1e0fa1e11e5ae11ac1accd07"}
Oct 31 04:54:59.207022 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:59.207003    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bca82d9baed57d8b7c7c622cda68f922d5bf47eb1e0fa1e11e5ae11ac1accd07"
Oct 31 04:54:59.207199 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:54:59.207185    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3fa8702-qff4w"
Oct 31 04:55:16.175043 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:55:16.174999    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:55:33.083904 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:33.082998    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wlt76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:55:33.083904 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:33.083081    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wlt76" podUID="c0469dfd-14dc-458a-9b3e-d5b0ecc25000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:55:43.525425 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.525383    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"]
Oct 31 04:55:43.525827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.525772    5415 cpu_manager.go:414] "containerMap: removing container" podUID="9ded1b2e-cb23-403a-bd07-3dcfaa0bea95" containerName="revoke-keys"
Oct 31 04:55:43.525827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.525783    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ded1b2e-cb23-403a-bd07-3dcfaa0bea95" containerName="revoke-keys"
Oct 31 04:55:43.525927 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.525892    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ded1b2e-cb23-403a-bd07-3dcfaa0bea95" containerName="revoke-keys"
Oct 31 04:55:43.529057 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.529027    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"
Oct 31 04:55:43.562433 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.562396    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4kj2\" (UniqueName: \"kubernetes.io/projected/35113590-55f3-4a69-a0bf-0d107fcf52b3-kube-api-access-l4kj2\") pod \"maas-api-revoke-sub-8833e2d9-qjblt\" (UID: \"35113590-55f3-4a69-a0bf-0d107fcf52b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"
Oct 31 04:55:43.562616 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.562499    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8833e2d9-qjblt\" (UID: \"35113590-55f3-4a69-a0bf-0d107fcf52b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"
Oct 31 04:55:43.562616 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.562555    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8833e2d9-qjblt\" (UID: \"35113590-55f3-4a69-a0bf-0d107fcf52b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"
Oct 31 04:55:43.578448 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.578409    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:55:43.625786 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.625747    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:55:43.627407 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.627382    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"]
Oct 31 04:55:43.663488 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.663449    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4kj2\" (UniqueName: \"kubernetes.io/projected/35113590-55f3-4a69-a0bf-0d107fcf52b3-kube-api-access-l4kj2\") pod \"maas-api-revoke-sub-8833e2d9-qjblt\" (UID: \"35113590-55f3-4a69-a0bf-0d107fcf52b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"
Oct 31 04:55:43.663688 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.663508    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8833e2d9-qjblt\" (UID: \"35113590-55f3-4a69-a0bf-0d107fcf52b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"
Oct 31 04:55:43.663688 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.663535    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8833e2d9-qjblt\" (UID: \"35113590-55f3-4a69-a0bf-0d107fcf52b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"
Oct 31 04:55:43.664302 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.664271    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8833e2d9-qjblt\" (UID: \"35113590-55f3-4a69-a0bf-0d107fcf52b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"
Oct 31 04:55:43.666043 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.666017    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8833e2d9-qjblt\" (UID: \"35113590-55f3-4a69-a0bf-0d107fcf52b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"
Oct 31 04:55:43.717618 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.717582    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4kj2\" (UniqueName: \"kubernetes.io/projected/35113590-55f3-4a69-a0bf-0d107fcf52b3-kube-api-access-l4kj2\") pod \"maas-api-revoke-sub-8833e2d9-qjblt\" (UID: \"35113590-55f3-4a69-a0bf-0d107fcf52b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"
Oct 31 04:55:43.842340 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:43.842301    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"
Oct 31 04:55:44.211846 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:44.211762    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"]
Oct 31 04:55:44.215634 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:55:44.215602    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35113590_55f3_4a69_a0bf_0d107fcf52b3.slice/crio-e14401fb220d09a34112f9de5027b4b37fe827e555495be918a323fc59952dfd WatchSource:0}: Error finding container e14401fb220d09a34112f9de5027b4b37fe827e555495be918a323fc59952dfd: Status 404 returned error can't find the container with id e14401fb220d09a34112f9de5027b4b37fe827e555495be918a323fc59952dfd
Oct 31 04:55:44.518733 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:44.518694    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt" event={"ID":"35113590-55f3-4a69-a0bf-0d107fcf52b3","Type":"ContainerStarted","Data":"e14401fb220d09a34112f9de5027b4b37fe827e555495be918a323fc59952dfd"}
Oct 31 04:55:45.525026 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:45.524987    5415 generic.go:356] "Generic (PLEG): container finished" podID="35113590-55f3-4a69-a0bf-0d107fcf52b3" containerID="444c36bffad0cf421a543e92a85fd7312ad9b88c7f4ed69e82abf667da029f39" exitCode=0
Oct 31 04:55:45.525454 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:45.525080    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt" event={"ID":"35113590-55f3-4a69-a0bf-0d107fcf52b3","Type":"ContainerDied","Data":"444c36bffad0cf421a543e92a85fd7312ad9b88c7f4ed69e82abf667da029f39"}
Oct 31 04:55:46.174721 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:55:46.174682    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:55:46.670440 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:46.670414    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"
Oct 31 04:55:46.796645 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:46.796567    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-openshift-ca\") pod \"35113590-55f3-4a69-a0bf-0d107fcf52b3\" (UID: \"35113590-55f3-4a69-a0bf-0d107fcf52b3\") "
Oct 31 04:55:46.796842 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:46.796649    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35113590-55f3-4a69-a0bf-0d107fcf52b3-kube-api-access-l4kj2\" (UniqueName: \"kubernetes.io/projected/35113590-55f3-4a69-a0bf-0d107fcf52b3-kube-api-access-l4kj2\") pod \"35113590-55f3-4a69-a0bf-0d107fcf52b3\" (UID: \"35113590-55f3-4a69-a0bf-0d107fcf52b3\") "
Oct 31 04:55:46.796842 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:46.796715    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-xks-ca\") pod \"35113590-55f3-4a69-a0bf-0d107fcf52b3\" (UID: \"35113590-55f3-4a69-a0bf-0d107fcf52b3\") "
Oct 31 04:55:46.797041 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:46.797015    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-openshift-ca" pod "35113590-55f3-4a69-a0bf-0d107fcf52b3" (UID: "35113590-55f3-4a69-a0bf-0d107fcf52b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:55:46.798933 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:46.798908    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-xks-ca" pod "35113590-55f3-4a69-a0bf-0d107fcf52b3" (UID: "35113590-55f3-4a69-a0bf-0d107fcf52b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:55:46.799065 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:46.799045    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35113590-55f3-4a69-a0bf-0d107fcf52b3-kube-api-access-l4kj2" pod "35113590-55f3-4a69-a0bf-0d107fcf52b3" (UID: "35113590-55f3-4a69-a0bf-0d107fcf52b3"). InnerVolumeSpecName "kube-api-access-l4kj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:55:46.897553 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:46.897516    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:55:46.897553 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:46.897547    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35113590-55f3-4a69-a0bf-0d107fcf52b3-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:55:46.897553 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:46.897560    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4kj2\" (UniqueName: \"kubernetes.io/projected/35113590-55f3-4a69-a0bf-0d107fcf52b3-kube-api-access-l4kj2\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:55:47.544525 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:47.544480    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt" event={"ID":"35113590-55f3-4a69-a0bf-0d107fcf52b3","Type":"ContainerDied","Data":"e14401fb220d09a34112f9de5027b4b37fe827e555495be918a323fc59952dfd"}
Oct 31 04:55:47.544716 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:47.544552    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e14401fb220d09a34112f9de5027b4b37fe827e555495be918a323fc59952dfd"
Oct 31 04:55:47.544716 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:47.544494    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8833e2d9-qjblt"
Oct 31 04:55:54.536560 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.536519    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"]
Oct 31 04:55:54.538104 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.538076    5415 cpu_manager.go:414] "containerMap: removing container" podUID="35113590-55f3-4a69-a0bf-0d107fcf52b3" containerName="revoke-keys"
Oct 31 04:55:54.538259 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.538246    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35113590-55f3-4a69-a0bf-0d107fcf52b3" containerName="revoke-keys"
Oct 31 04:55:54.538574 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.538559    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="35113590-55f3-4a69-a0bf-0d107fcf52b3" containerName="revoke-keys"
Oct 31 04:55:54.543131 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.543098    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"
Oct 31 04:55:54.593147 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.593115    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"]
Oct 31 04:55:54.638436 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.638399    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:55:54.638622 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.638457    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:55:54.676668 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.676615    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e5c068e-98rkk\" (UID: \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"
Oct 31 04:55:54.676860 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.676761    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ctp\" (UniqueName: \"kubernetes.io/projected/a1a1e27a-db48-4b9d-8775-f0751efb35f6-kube-api-access-n9ctp\") pod \"maas-api-revoke-sub-0e5c068e-98rkk\" (UID: \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"
Oct 31 04:55:54.676860 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.676848    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e5c068e-98rkk\" (UID: \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"
Oct 31 04:55:54.778132 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.778091    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ctp\" (UniqueName: \"kubernetes.io/projected/a1a1e27a-db48-4b9d-8775-f0751efb35f6-kube-api-access-n9ctp\") pod \"maas-api-revoke-sub-0e5c068e-98rkk\" (UID: \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"
Oct 31 04:55:54.778323 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.778144    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e5c068e-98rkk\" (UID: \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"
Oct 31 04:55:54.778323 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.778195    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e5c068e-98rkk\" (UID: \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"
Oct 31 04:55:54.778900 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.778847    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e5c068e-98rkk\" (UID: \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"
Oct 31 04:55:54.780584 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.780564    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e5c068e-98rkk\" (UID: \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"
Oct 31 04:55:54.832328 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.832247    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ctp\" (UniqueName: \"kubernetes.io/projected/a1a1e27a-db48-4b9d-8775-f0751efb35f6-kube-api-access-n9ctp\") pod \"maas-api-revoke-sub-0e5c068e-98rkk\" (UID: \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"
Oct 31 04:55:54.861148 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:54.861109    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"
Oct 31 04:55:55.229023 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:55.229000    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"]
Oct 31 04:55:55.231961 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:55:55.231852    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a1e27a_db48_4b9d_8775_f0751efb35f6.slice/crio-950052db1db146d4405d322afb76655905e6c58b8061baaf12410cf84bfb6266 WatchSource:0}: Error finding container 950052db1db146d4405d322afb76655905e6c58b8061baaf12410cf84bfb6266: Status 404 returned error can't find the container with id 950052db1db146d4405d322afb76655905e6c58b8061baaf12410cf84bfb6266
Oct 31 04:55:55.587411 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:55.587375    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk" event={"ID":"a1a1e27a-db48-4b9d-8775-f0751efb35f6","Type":"ContainerStarted","Data":"f66e1f7b11bb6af10c5f1b560717edb7765453a7d7e4f688b1eec5f957ef2500"}
Oct 31 04:55:55.587818 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:55.587428    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk" event={"ID":"a1a1e27a-db48-4b9d-8775-f0751efb35f6","Type":"ContainerStarted","Data":"950052db1db146d4405d322afb76655905e6c58b8061baaf12410cf84bfb6266"}
Oct 31 04:55:56.594086 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:56.594054    5415 generic.go:356] "Generic (PLEG): container finished" podID="a1a1e27a-db48-4b9d-8775-f0751efb35f6" containerID="f66e1f7b11bb6af10c5f1b560717edb7765453a7d7e4f688b1eec5f957ef2500" exitCode=0
Oct 31 04:55:56.594520 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:56.594129    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk" event={"ID":"a1a1e27a-db48-4b9d-8775-f0751efb35f6","Type":"ContainerDied","Data":"f66e1f7b11bb6af10c5f1b560717edb7765453a7d7e4f688b1eec5f957ef2500"}
Oct 31 04:55:57.750045 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:57.750020    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"
Oct 31 04:55:57.907726 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:57.907632    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-xks-ca\") pod \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\" (UID: \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\") "
Oct 31 04:55:57.907930 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:57.907784    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a1e27a-db48-4b9d-8775-f0751efb35f6-kube-api-access-n9ctp\" (UniqueName: \"kubernetes.io/projected/a1a1e27a-db48-4b9d-8775-f0751efb35f6-kube-api-access-n9ctp\") pod \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\" (UID: \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\") "
Oct 31 04:55:57.907930 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:57.907852    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-openshift-ca\") pod \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\" (UID: \"a1a1e27a-db48-4b9d-8775-f0751efb35f6\") "
Oct 31 04:55:57.908160 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:57.908137    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-openshift-ca" pod "a1a1e27a-db48-4b9d-8775-f0751efb35f6" (UID: "a1a1e27a-db48-4b9d-8775-f0751efb35f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:55:57.909729 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:57.909707    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-xks-ca" pod "a1a1e27a-db48-4b9d-8775-f0751efb35f6" (UID: "a1a1e27a-db48-4b9d-8775-f0751efb35f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:55:57.909827 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:57.909812    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a1e27a-db48-4b9d-8775-f0751efb35f6-kube-api-access-n9ctp" pod "a1a1e27a-db48-4b9d-8775-f0751efb35f6" (UID: "a1a1e27a-db48-4b9d-8775-f0751efb35f6"). InnerVolumeSpecName "kube-api-access-n9ctp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:55:58.009310 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:58.009267    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9ctp\" (UniqueName: \"kubernetes.io/projected/a1a1e27a-db48-4b9d-8775-f0751efb35f6-kube-api-access-n9ctp\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:55:58.009310 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:58.009301    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:55:58.009310 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:58.009313    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a1e27a-db48-4b9d-8775-f0751efb35f6-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:55:58.607667 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:58.607615    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk" event={"ID":"a1a1e27a-db48-4b9d-8775-f0751efb35f6","Type":"ContainerDied","Data":"950052db1db146d4405d322afb76655905e6c58b8061baaf12410cf84bfb6266"}
Oct 31 04:55:58.607906 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:58.607686    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="950052db1db146d4405d322afb76655905e6c58b8061baaf12410cf84bfb6266"
Oct 31 04:55:58.607906 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:55:58.607645    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5c068e-98rkk"
Oct 31 04:56:05.181964 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.181932    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"]
Oct 31 04:56:05.182477 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.182328    5415 cpu_manager.go:414] "containerMap: removing container" podUID="a1a1e27a-db48-4b9d-8775-f0751efb35f6" containerName="revoke-keys"
Oct 31 04:56:05.182477 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.182339    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a1e27a-db48-4b9d-8775-f0751efb35f6" containerName="revoke-keys"
Oct 31 04:56:05.182477 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.182402    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a1e27a-db48-4b9d-8775-f0751efb35f6" containerName="revoke-keys"
Oct 31 04:56:05.185386 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.185362    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"
Oct 31 04:56:05.240204 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.240167    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:56:05.240204 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.240187    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:56:05.242132 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.242102    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"]
Oct 31 04:56:05.280801 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.280767    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd228\" (UniqueName: \"kubernetes.io/projected/a27d5840-4f8e-46e6-82b5-41c09686e1c6-kube-api-access-pd228\") pod \"maas-api-revoke-sub-52fd21f9-krtfh\" (UID: \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"
Oct 31 04:56:05.281003 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.280925    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52fd21f9-krtfh\" (UID: \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"
Oct 31 04:56:05.281003 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.280973    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52fd21f9-krtfh\" (UID: \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"
Oct 31 04:56:05.381800 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.381759    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52fd21f9-krtfh\" (UID: \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"
Oct 31 04:56:05.382029 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.381817    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52fd21f9-krtfh\" (UID: \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"
Oct 31 04:56:05.382029 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.381863    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd228\" (UniqueName: \"kubernetes.io/projected/a27d5840-4f8e-46e6-82b5-41c09686e1c6-kube-api-access-pd228\") pod \"maas-api-revoke-sub-52fd21f9-krtfh\" (UID: \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"
Oct 31 04:56:05.382664 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.382639    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52fd21f9-krtfh\" (UID: \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"
Oct 31 04:56:05.384301 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.384279    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52fd21f9-krtfh\" (UID: \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"
Oct 31 04:56:05.438268 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.438196    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd228\" (UniqueName: \"kubernetes.io/projected/a27d5840-4f8e-46e6-82b5-41c09686e1c6-kube-api-access-pd228\") pod \"maas-api-revoke-sub-52fd21f9-krtfh\" (UID: \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"
Oct 31 04:56:05.500462 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.500426    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"
Oct 31 04:56:05.869198 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:05.869175    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"]
Oct 31 04:56:05.871745 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:56:05.871716    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27d5840_4f8e_46e6_82b5_41c09686e1c6.slice/crio-539c44cb97f351049ccc75670af1237a8a446445c425ee4773b3334977235fbd WatchSource:0}: Error finding container 539c44cb97f351049ccc75670af1237a8a446445c425ee4773b3334977235fbd: Status 404 returned error can't find the container with id 539c44cb97f351049ccc75670af1237a8a446445c425ee4773b3334977235fbd
Oct 31 04:56:06.655995 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:06.655954    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh" event={"ID":"a27d5840-4f8e-46e6-82b5-41c09686e1c6","Type":"ContainerStarted","Data":"7204042a141132400b5de2ec590dc48be62342aa53d71bcac0b9d88113a7d346"}
Oct 31 04:56:06.656420 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:06.656009    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh" event={"ID":"a27d5840-4f8e-46e6-82b5-41c09686e1c6","Type":"ContainerStarted","Data":"539c44cb97f351049ccc75670af1237a8a446445c425ee4773b3334977235fbd"}
Oct 31 04:56:06.764456 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:06.764405    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh" podStartSLOduration=1.764389553 podStartE2EDuration="1.764389553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:56:06.76308249 +0000 UTC m=+6508.323562238" watchObservedRunningTime="2026-10-31 04:56:06.764389553 +0000 UTC m=+6508.324869284"
Oct 31 04:56:07.662706 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:07.662671    5415 generic.go:356] "Generic (PLEG): container finished" podID="a27d5840-4f8e-46e6-82b5-41c09686e1c6" containerID="7204042a141132400b5de2ec590dc48be62342aa53d71bcac0b9d88113a7d346" exitCode=0
Oct 31 04:56:07.663144 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:07.662722    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh" event={"ID":"a27d5840-4f8e-46e6-82b5-41c09686e1c6","Type":"ContainerDied","Data":"7204042a141132400b5de2ec590dc48be62342aa53d71bcac0b9d88113a7d346"}
Oct 31 04:56:08.802806 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:08.802778    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"
Oct 31 04:56:08.914542 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:08.914507    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-openshift-ca\") pod \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\" (UID: \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\") "
Oct 31 04:56:08.914746 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:08.914560    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-xks-ca\") pod \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\" (UID: \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\") "
Oct 31 04:56:08.914746 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:08.914616    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27d5840-4f8e-46e6-82b5-41c09686e1c6-kube-api-access-pd228\" (UniqueName: \"kubernetes.io/projected/a27d5840-4f8e-46e6-82b5-41c09686e1c6-kube-api-access-pd228\") pod \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\" (UID: \"a27d5840-4f8e-46e6-82b5-41c09686e1c6\") "
Oct 31 04:56:08.915006 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:08.914958    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-openshift-ca" pod "a27d5840-4f8e-46e6-82b5-41c09686e1c6" (UID: "a27d5840-4f8e-46e6-82b5-41c09686e1c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:56:08.916994 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:08.916958    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27d5840-4f8e-46e6-82b5-41c09686e1c6-kube-api-access-pd228" pod "a27d5840-4f8e-46e6-82b5-41c09686e1c6" (UID: "a27d5840-4f8e-46e6-82b5-41c09686e1c6"). InnerVolumeSpecName "kube-api-access-pd228". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:56:08.917119 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:08.917015    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-xks-ca" pod "a27d5840-4f8e-46e6-82b5-41c09686e1c6" (UID: "a27d5840-4f8e-46e6-82b5-41c09686e1c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:56:09.015844 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:09.015805    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:09.015844 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:09.015840    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a27d5840-4f8e-46e6-82b5-41c09686e1c6-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:09.015844 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:09.015850    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd228\" (UniqueName: \"kubernetes.io/projected/a27d5840-4f8e-46e6-82b5-41c09686e1c6-kube-api-access-pd228\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:09.673321 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:09.673282    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh" event={"ID":"a27d5840-4f8e-46e6-82b5-41c09686e1c6","Type":"ContainerDied","Data":"539c44cb97f351049ccc75670af1237a8a446445c425ee4773b3334977235fbd"}
Oct 31 04:56:09.673321 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:09.673323    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fd21f9-krtfh"
Oct 31 04:56:09.673563 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:09.673347    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="539c44cb97f351049ccc75670af1237a8a446445c425ee4773b3334977235fbd"
Oct 31 04:56:16.142340 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.142307    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"]
Oct 31 04:56:16.142789 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.142729    5415 cpu_manager.go:414] "containerMap: removing container" podUID="a27d5840-4f8e-46e6-82b5-41c09686e1c6" containerName="revoke-keys"
Oct 31 04:56:16.142789 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.142742    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a27d5840-4f8e-46e6-82b5-41c09686e1c6" containerName="revoke-keys"
Oct 31 04:56:16.142867 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.142822    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="a27d5840-4f8e-46e6-82b5-41c09686e1c6" containerName="revoke-keys"
Oct 31 04:56:16.146465 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.146442    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"
Oct 31 04:56:16.174432 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:56:16.174399    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:56:16.243231 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.243193    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:56:16.243231 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.243208    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:56:16.245099 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.245075    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"]
Oct 31 04:56:16.289102 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.289062    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22bjz\" (UniqueName: \"kubernetes.io/projected/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-kube-api-access-22bjz\") pod \"maas-api-revoke-sub-ff6aa3d8-wt5pw\" (UID: \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"
Oct 31 04:56:16.289102 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.289110    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff6aa3d8-wt5pw\" (UID: \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"
Oct 31 04:56:16.289373 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.289214    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff6aa3d8-wt5pw\" (UID: \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"
Oct 31 04:56:16.390096 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.390053    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff6aa3d8-wt5pw\" (UID: \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"
Oct 31 04:56:16.390281 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.390152    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22bjz\" (UniqueName: \"kubernetes.io/projected/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-kube-api-access-22bjz\") pod \"maas-api-revoke-sub-ff6aa3d8-wt5pw\" (UID: \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"
Oct 31 04:56:16.390281 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.390181    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff6aa3d8-wt5pw\" (UID: \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"
Oct 31 04:56:16.390836 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.390818    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff6aa3d8-wt5pw\" (UID: \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"
Oct 31 04:56:16.392568 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.392523    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff6aa3d8-wt5pw\" (UID: \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"
Oct 31 04:56:16.444680 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.444645    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22bjz\" (UniqueName: \"kubernetes.io/projected/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-kube-api-access-22bjz\") pod \"maas-api-revoke-sub-ff6aa3d8-wt5pw\" (UID: \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"
Oct 31 04:56:16.459521 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.459483    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"
Oct 31 04:56:16.823744 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:16.823717    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"]
Oct 31 04:56:16.826801 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:56:16.826771    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd0bcdf_26a5_4045_826b_54e5e77dc3a1.slice/crio-e53fd563dd557243eac5c210a8d2f4fc9e7c71571f646cc591563407c3f88d1c WatchSource:0}: Error finding container e53fd563dd557243eac5c210a8d2f4fc9e7c71571f646cc591563407c3f88d1c: Status 404 returned error can't find the container with id e53fd563dd557243eac5c210a8d2f4fc9e7c71571f646cc591563407c3f88d1c
Oct 31 04:56:17.712714 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:17.712681    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw" event={"ID":"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1","Type":"ContainerStarted","Data":"0d369b9bab81e8b2a0a44d9b85e077322d2b44ffc7afcd9f372199f4f44203c7"}
Oct 31 04:56:17.713288 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:17.712736    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw" event={"ID":"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1","Type":"ContainerStarted","Data":"e53fd563dd557243eac5c210a8d2f4fc9e7c71571f646cc591563407c3f88d1c"}
Oct 31 04:56:17.821007 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:17.820954    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw" podStartSLOduration=1.820936416 podStartE2EDuration="1.820936416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:56:17.819186377 +0000 UTC m=+6519.379666110" watchObservedRunningTime="2026-10-31 04:56:17.820936416 +0000 UTC m=+6519.381416149"
Oct 31 04:56:18.720923 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:18.720857    5415 generic.go:356] "Generic (PLEG): container finished" podID="2bd0bcdf-26a5-4045-826b-54e5e77dc3a1" containerID="0d369b9bab81e8b2a0a44d9b85e077322d2b44ffc7afcd9f372199f4f44203c7" exitCode=0
Oct 31 04:56:18.721383 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:18.720945    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw" event={"ID":"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1","Type":"ContainerDied","Data":"0d369b9bab81e8b2a0a44d9b85e077322d2b44ffc7afcd9f372199f4f44203c7"}
Oct 31 04:56:20.141677 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.141647    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"
Oct 31 04:56:20.233350 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.233304    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-kube-api-access-22bjz\" (UniqueName: \"kubernetes.io/projected/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-kube-api-access-22bjz\") pod \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\" (UID: \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\") "
Oct 31 04:56:20.233572 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.233413    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-xks-ca\") pod \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\" (UID: \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\") "
Oct 31 04:56:20.233572 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.233436    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-openshift-ca\") pod \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\" (UID: \"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1\") "
Oct 31 04:56:20.233967 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.233945    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-openshift-ca" pod "2bd0bcdf-26a5-4045-826b-54e5e77dc3a1" (UID: "2bd0bcdf-26a5-4045-826b-54e5e77dc3a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:56:20.235405 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.235381    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-kube-api-access-22bjz" pod "2bd0bcdf-26a5-4045-826b-54e5e77dc3a1" (UID: "2bd0bcdf-26a5-4045-826b-54e5e77dc3a1"). InnerVolumeSpecName "kube-api-access-22bjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:56:20.235561 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.235541    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-xks-ca" pod "2bd0bcdf-26a5-4045-826b-54e5e77dc3a1" (UID: "2bd0bcdf-26a5-4045-826b-54e5e77dc3a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:56:20.256116 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.256003    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5smg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:56:20.256116 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.256074    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5smg5" podUID="1cfc6a64-edfa-4e51-83b5-f38f438f1877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:56:20.334832 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.334804    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:20.334832 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.334833    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:20.335099 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.334842    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22bjz\" (UniqueName: \"kubernetes.io/projected/2bd0bcdf-26a5-4045-826b-54e5e77dc3a1-kube-api-access-22bjz\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:20.735555 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.735520    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw"
Oct 31 04:56:20.735555 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.735544    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6aa3d8-wt5pw" event={"ID":"2bd0bcdf-26a5-4045-826b-54e5e77dc3a1","Type":"ContainerDied","Data":"e53fd563dd557243eac5c210a8d2f4fc9e7c71571f646cc591563407c3f88d1c"}
Oct 31 04:56:20.735783 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:20.735603    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e53fd563dd557243eac5c210a8d2f4fc9e7c71571f646cc591563407c3f88d1c"
Oct 31 04:56:27.084015 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.083972    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"]
Oct 31 04:56:27.084435 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.084371    5415 cpu_manager.go:414] "containerMap: removing container" podUID="2bd0bcdf-26a5-4045-826b-54e5e77dc3a1" containerName="revoke-keys"
Oct 31 04:56:27.084435 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.084382    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd0bcdf-26a5-4045-826b-54e5e77dc3a1" containerName="revoke-keys"
Oct 31 04:56:27.084541 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.084485    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd0bcdf-26a5-4045-826b-54e5e77dc3a1" containerName="revoke-keys"
Oct 31 04:56:27.087550 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.087516    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"
Oct 31 04:56:27.184551 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.184511    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:56:27.184551 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.184517    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:56:27.186638 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.186609    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"]
Oct 31 04:56:27.201243 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.201214    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efb1238d-45clm\" (UID: \"4f7d679c-f688-4470-9d8d-71f27139997d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"
Oct 31 04:56:27.201401 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.201344    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbf7b\" (UniqueName: \"kubernetes.io/projected/4f7d679c-f688-4470-9d8d-71f27139997d-kube-api-access-kbf7b\") pod \"maas-api-revoke-sub-efb1238d-45clm\" (UID: \"4f7d679c-f688-4470-9d8d-71f27139997d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"
Oct 31 04:56:27.201480 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.201460    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efb1238d-45clm\" (UID: \"4f7d679c-f688-4470-9d8d-71f27139997d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"
Oct 31 04:56:27.302272 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.302232    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efb1238d-45clm\" (UID: \"4f7d679c-f688-4470-9d8d-71f27139997d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"
Oct 31 04:56:27.302460 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.302330    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbf7b\" (UniqueName: \"kubernetes.io/projected/4f7d679c-f688-4470-9d8d-71f27139997d-kube-api-access-kbf7b\") pod \"maas-api-revoke-sub-efb1238d-45clm\" (UID: \"4f7d679c-f688-4470-9d8d-71f27139997d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"
Oct 31 04:56:27.302460 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.302416    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efb1238d-45clm\" (UID: \"4f7d679c-f688-4470-9d8d-71f27139997d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"
Oct 31 04:56:27.303138 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.303114    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efb1238d-45clm\" (UID: \"4f7d679c-f688-4470-9d8d-71f27139997d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"
Oct 31 04:56:27.304780 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.304756    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efb1238d-45clm\" (UID: \"4f7d679c-f688-4470-9d8d-71f27139997d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"
Oct 31 04:56:27.356803 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.356728    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbf7b\" (UniqueName: \"kubernetes.io/projected/4f7d679c-f688-4470-9d8d-71f27139997d-kube-api-access-kbf7b\") pod \"maas-api-revoke-sub-efb1238d-45clm\" (UID: \"4f7d679c-f688-4470-9d8d-71f27139997d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"
Oct 31 04:56:27.400716 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.400679    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"
Oct 31 04:56:27.766066 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:27.765993    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"]
Oct 31 04:56:27.768810 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:56:27.768774    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7d679c_f688_4470_9d8d_71f27139997d.slice/crio-6cd05c19a097359393cd7a24fc3af15fe6efd5f3cf2778ce0a5bd3d4dd160efd WatchSource:0}: Error finding container 6cd05c19a097359393cd7a24fc3af15fe6efd5f3cf2778ce0a5bd3d4dd160efd: Status 404 returned error can't find the container with id 6cd05c19a097359393cd7a24fc3af15fe6efd5f3cf2778ce0a5bd3d4dd160efd
Oct 31 04:56:28.779902 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:28.779804    5415 generic.go:356] "Generic (PLEG): container finished" podID="4f7d679c-f688-4470-9d8d-71f27139997d" containerID="72fd9299566b71c457683d1e25995e454676399a1c117515d93dd8f97075df6a" exitCode=0
Oct 31 04:56:28.779902 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:28.779842    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm" event={"ID":"4f7d679c-f688-4470-9d8d-71f27139997d","Type":"ContainerDied","Data":"72fd9299566b71c457683d1e25995e454676399a1c117515d93dd8f97075df6a"}
Oct 31 04:56:28.780317 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:28.779904    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm" event={"ID":"4f7d679c-f688-4470-9d8d-71f27139997d","Type":"ContainerStarted","Data":"6cd05c19a097359393cd7a24fc3af15fe6efd5f3cf2778ce0a5bd3d4dd160efd"}
Oct 31 04:56:29.948596 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:29.948566    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"
Oct 31 04:56:30.029063 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:30.029025    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-openshift-ca\") pod \"4f7d679c-f688-4470-9d8d-71f27139997d\" (UID: \"4f7d679c-f688-4470-9d8d-71f27139997d\") "
Oct 31 04:56:30.029063 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:30.029079    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7d679c-f688-4470-9d8d-71f27139997d-kube-api-access-kbf7b\" (UniqueName: \"kubernetes.io/projected/4f7d679c-f688-4470-9d8d-71f27139997d-kube-api-access-kbf7b\") pod \"4f7d679c-f688-4470-9d8d-71f27139997d\" (UID: \"4f7d679c-f688-4470-9d8d-71f27139997d\") "
Oct 31 04:56:30.029434 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:30.029173    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-xks-ca\") pod \"4f7d679c-f688-4470-9d8d-71f27139997d\" (UID: \"4f7d679c-f688-4470-9d8d-71f27139997d\") "
Oct 31 04:56:30.029524 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:30.029495    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-openshift-ca" pod "4f7d679c-f688-4470-9d8d-71f27139997d" (UID: "4f7d679c-f688-4470-9d8d-71f27139997d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:56:30.032105 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:30.032068    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7d679c-f688-4470-9d8d-71f27139997d-kube-api-access-kbf7b" pod "4f7d679c-f688-4470-9d8d-71f27139997d" (UID: "4f7d679c-f688-4470-9d8d-71f27139997d"). InnerVolumeSpecName "kube-api-access-kbf7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:56:30.032263 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:30.032075    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-xks-ca" pod "4f7d679c-f688-4470-9d8d-71f27139997d" (UID: "4f7d679c-f688-4470-9d8d-71f27139997d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:56:30.130698 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:30.130659    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:30.130698 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:30.130693    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d679c-f688-4470-9d8d-71f27139997d-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:30.130698 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:30.130703    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbf7b\" (UniqueName: \"kubernetes.io/projected/4f7d679c-f688-4470-9d8d-71f27139997d-kube-api-access-kbf7b\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:30.819904 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:30.817132    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm" event={"ID":"4f7d679c-f688-4470-9d8d-71f27139997d","Type":"ContainerDied","Data":"6cd05c19a097359393cd7a24fc3af15fe6efd5f3cf2778ce0a5bd3d4dd160efd"}
Oct 31 04:56:30.819904 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:30.817204    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cd05c19a097359393cd7a24fc3af15fe6efd5f3cf2778ce0a5bd3d4dd160efd"
Oct 31 04:56:30.819904 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:30.817283    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb1238d-45clm"
Oct 31 04:56:38.041319 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.041271    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"]
Oct 31 04:56:38.041910 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.041726    5415 cpu_manager.go:414] "containerMap: removing container" podUID="4f7d679c-f688-4470-9d8d-71f27139997d" containerName="revoke-keys"
Oct 31 04:56:38.041910 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.041740    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7d679c-f688-4470-9d8d-71f27139997d" containerName="revoke-keys"
Oct 31 04:56:38.041910 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.041818    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7d679c-f688-4470-9d8d-71f27139997d" containerName="revoke-keys"
Oct 31 04:56:38.045340 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.045320    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"
Oct 31 04:56:38.095139 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.095109    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:56:38.095321 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.095164    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:56:38.109326 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.109280    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a94ae629-xzgjb\" (UID: \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"
Oct 31 04:56:38.109505 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.109354    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnk75\" (UniqueName: \"kubernetes.io/projected/e5cfac24-eb41-424d-94a1-4bc541bf90a2-kube-api-access-vnk75\") pod \"maas-api-revoke-sub-a94ae629-xzgjb\" (UID: \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"
Oct 31 04:56:38.109559 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.109497    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a94ae629-xzgjb\" (UID: \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"
Oct 31 04:56:38.144532 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.144496    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"]
Oct 31 04:56:38.210297 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.210259    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnk75\" (UniqueName: \"kubernetes.io/projected/e5cfac24-eb41-424d-94a1-4bc541bf90a2-kube-api-access-vnk75\") pod \"maas-api-revoke-sub-a94ae629-xzgjb\" (UID: \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"
Oct 31 04:56:38.210510 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.210348    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a94ae629-xzgjb\" (UID: \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"
Oct 31 04:56:38.210510 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.210389    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a94ae629-xzgjb\" (UID: \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"
Oct 31 04:56:38.211085 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.211066    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a94ae629-xzgjb\" (UID: \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"
Oct 31 04:56:38.212856 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.212830    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a94ae629-xzgjb\" (UID: \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"
Oct 31 04:56:38.264605 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.264565    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnk75\" (UniqueName: \"kubernetes.io/projected/e5cfac24-eb41-424d-94a1-4bc541bf90a2-kube-api-access-vnk75\") pod \"maas-api-revoke-sub-a94ae629-xzgjb\" (UID: \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"
Oct 31 04:56:38.358857 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.358818    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"
Oct 31 04:56:38.724234 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.724208    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"]
Oct 31 04:56:38.727052 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:56:38.727003    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cfac24_eb41_424d_94a1_4bc541bf90a2.slice/crio-5896e5ccb8611fe8b41c1334d92c1d9b3a6cb68164e9915acb4d4017c209ae45 WatchSource:0}: Error finding container 5896e5ccb8611fe8b41c1334d92c1d9b3a6cb68164e9915acb4d4017c209ae45: Status 404 returned error can't find the container with id 5896e5ccb8611fe8b41c1334d92c1d9b3a6cb68164e9915acb4d4017c209ae45
Oct 31 04:56:38.862457 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:38.862420    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb" event={"ID":"e5cfac24-eb41-424d-94a1-4bc541bf90a2","Type":"ContainerStarted","Data":"5896e5ccb8611fe8b41c1334d92c1d9b3a6cb68164e9915acb4d4017c209ae45"}
Oct 31 04:56:39.869548 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:39.869508    5415 generic.go:356] "Generic (PLEG): container finished" podID="e5cfac24-eb41-424d-94a1-4bc541bf90a2" containerID="0d9bc82a9468c1b4ca7f389f7f8f0c2bfc93c8bc797a0f33c1a5962ee03b811a" exitCode=0
Oct 31 04:56:39.869982 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:39.869600    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb" event={"ID":"e5cfac24-eb41-424d-94a1-4bc541bf90a2","Type":"ContainerDied","Data":"0d9bc82a9468c1b4ca7f389f7f8f0c2bfc93c8bc797a0f33c1a5962ee03b811a"}
Oct 31 04:56:41.073737 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:41.073371    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"
Oct 31 04:56:41.137432 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:41.137322    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-openshift-ca\") pod \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\" (UID: \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\") "
Oct 31 04:56:41.137432 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:41.137434    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cfac24-eb41-424d-94a1-4bc541bf90a2-kube-api-access-vnk75\" (UniqueName: \"kubernetes.io/projected/e5cfac24-eb41-424d-94a1-4bc541bf90a2-kube-api-access-vnk75\") pod \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\" (UID: \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\") "
Oct 31 04:56:41.137696 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:41.137595    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-xks-ca\") pod \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\" (UID: \"e5cfac24-eb41-424d-94a1-4bc541bf90a2\") "
Oct 31 04:56:41.137776 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:41.137748    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-openshift-ca" pod "e5cfac24-eb41-424d-94a1-4bc541bf90a2" (UID: "e5cfac24-eb41-424d-94a1-4bc541bf90a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:56:41.137983 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:41.137963    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:41.139559 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:41.139536    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cfac24-eb41-424d-94a1-4bc541bf90a2-kube-api-access-vnk75" pod "e5cfac24-eb41-424d-94a1-4bc541bf90a2" (UID: "e5cfac24-eb41-424d-94a1-4bc541bf90a2"). InnerVolumeSpecName "kube-api-access-vnk75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:56:41.139669 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:41.139590    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-xks-ca" pod "e5cfac24-eb41-424d-94a1-4bc541bf90a2" (UID: "e5cfac24-eb41-424d-94a1-4bc541bf90a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:56:41.238671 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:41.238626    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5cfac24-eb41-424d-94a1-4bc541bf90a2-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:41.238915 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:41.238706    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnk75\" (UniqueName: \"kubernetes.io/projected/e5cfac24-eb41-424d-94a1-4bc541bf90a2-kube-api-access-vnk75\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:41.905288 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:41.905243    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb" event={"ID":"e5cfac24-eb41-424d-94a1-4bc541bf90a2","Type":"ContainerDied","Data":"5896e5ccb8611fe8b41c1334d92c1d9b3a6cb68164e9915acb4d4017c209ae45"}
Oct 31 04:56:41.905607 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:41.905585    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5896e5ccb8611fe8b41c1334d92c1d9b3a6cb68164e9915acb4d4017c209ae45"
Oct 31 04:56:41.905794 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:41.905779    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94ae629-xzgjb"
Oct 31 04:56:42.564905 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:42.560173    5415 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bwzrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:56:42.564905 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:42.560308    5415 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bwzrc" podUID="c9d3bfd0-545c-4be0-92ae-30a7c68138ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:56:46.175331 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:56:46.175293    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:56:48.761649 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.761613    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"]
Oct 31 04:56:48.762078 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.762039    5415 cpu_manager.go:414] "containerMap: removing container" podUID="e5cfac24-eb41-424d-94a1-4bc541bf90a2" containerName="revoke-keys"
Oct 31 04:56:48.762078 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.762050    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cfac24-eb41-424d-94a1-4bc541bf90a2" containerName="revoke-keys"
Oct 31 04:56:48.762162 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.762150    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cfac24-eb41-424d-94a1-4bc541bf90a2" containerName="revoke-keys"
Oct 31 04:56:48.770427 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.770395    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"
Oct 31 04:56:48.822412 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.822378    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"]
Oct 31 04:56:48.829958 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.829856    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b040a6ad-jlj8b\" (UID: \"371cbd89-b9fb-450b-8d53-477f21a0f677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"
Oct 31 04:56:48.830154 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.830019    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b040a6ad-jlj8b\" (UID: \"371cbd89-b9fb-450b-8d53-477f21a0f677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"
Oct 31 04:56:48.830154 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.830086    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmzk2\" (UniqueName: \"kubernetes.io/projected/371cbd89-b9fb-450b-8d53-477f21a0f677-kube-api-access-cmzk2\") pod \"maas-api-revoke-sub-b040a6ad-jlj8b\" (UID: \"371cbd89-b9fb-450b-8d53-477f21a0f677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"
Oct 31 04:56:48.867069 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.867027    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:56:48.867069 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.867063    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:56:48.931460 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.931422    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzk2\" (UniqueName: \"kubernetes.io/projected/371cbd89-b9fb-450b-8d53-477f21a0f677-kube-api-access-cmzk2\") pod \"maas-api-revoke-sub-b040a6ad-jlj8b\" (UID: \"371cbd89-b9fb-450b-8d53-477f21a0f677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"
Oct 31 04:56:48.931668 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.931495    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b040a6ad-jlj8b\" (UID: \"371cbd89-b9fb-450b-8d53-477f21a0f677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"
Oct 31 04:56:48.931668 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.931532    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b040a6ad-jlj8b\" (UID: \"371cbd89-b9fb-450b-8d53-477f21a0f677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"
Oct 31 04:56:48.932191 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.932169    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b040a6ad-jlj8b\" (UID: \"371cbd89-b9fb-450b-8d53-477f21a0f677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"
Oct 31 04:56:48.934041 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.934017    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b040a6ad-jlj8b\" (UID: \"371cbd89-b9fb-450b-8d53-477f21a0f677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"
Oct 31 04:56:48.985405 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:48.985368    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzk2\" (UniqueName: \"kubernetes.io/projected/371cbd89-b9fb-450b-8d53-477f21a0f677-kube-api-access-cmzk2\") pod \"maas-api-revoke-sub-b040a6ad-jlj8b\" (UID: \"371cbd89-b9fb-450b-8d53-477f21a0f677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"
Oct 31 04:56:49.084171 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:49.084136    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"
Oct 31 04:56:49.454337 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:49.454311    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"]
Oct 31 04:56:49.456870 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:56:49.456837    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371cbd89_b9fb_450b_8d53_477f21a0f677.slice/crio-94dac5aafa4b6f702939a208826e9979d6646138b4a0ba1969086e46fa29238b WatchSource:0}: Error finding container 94dac5aafa4b6f702939a208826e9979d6646138b4a0ba1969086e46fa29238b: Status 404 returned error can't find the container with id 94dac5aafa4b6f702939a208826e9979d6646138b4a0ba1969086e46fa29238b
Oct 31 04:56:49.947099 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:49.947063    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b" event={"ID":"371cbd89-b9fb-450b-8d53-477f21a0f677","Type":"ContainerStarted","Data":"dddb90493efb9e34cb9427153c6b223e013f5d5d86e96078e6e2986c9049cb2b"}
Oct 31 04:56:49.947508 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:49.947130    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b" event={"ID":"371cbd89-b9fb-450b-8d53-477f21a0f677","Type":"ContainerStarted","Data":"94dac5aafa4b6f702939a208826e9979d6646138b4a0ba1969086e46fa29238b"}
Oct 31 04:56:50.057003 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:50.056804    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b" podStartSLOduration=2.056788483 podStartE2EDuration="2.056788483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:56:50.054739327 +0000 UTC m=+6551.615219059" watchObservedRunningTime="2026-10-31 04:56:50.056788483 +0000 UTC m=+6551.617268210"
Oct 31 04:56:50.953518 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:50.953482    5415 generic.go:356] "Generic (PLEG): container finished" podID="371cbd89-b9fb-450b-8d53-477f21a0f677" containerID="dddb90493efb9e34cb9427153c6b223e013f5d5d86e96078e6e2986c9049cb2b" exitCode=0
Oct 31 04:56:50.953942 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:50.953564    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b" event={"ID":"371cbd89-b9fb-450b-8d53-477f21a0f677","Type":"ContainerDied","Data":"dddb90493efb9e34cb9427153c6b223e013f5d5d86e96078e6e2986c9049cb2b"}
Oct 31 04:56:52.210527 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:52.210112    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"
Oct 31 04:56:52.275339 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:52.275282    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-xks-ca\") pod \"371cbd89-b9fb-450b-8d53-477f21a0f677\" (UID: \"371cbd89-b9fb-450b-8d53-477f21a0f677\") "
Oct 31 04:56:52.275695 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:52.275669    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-openshift-ca\") pod \"371cbd89-b9fb-450b-8d53-477f21a0f677\" (UID: \"371cbd89-b9fb-450b-8d53-477f21a0f677\") "
Oct 31 04:56:52.275907 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:52.275860    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371cbd89-b9fb-450b-8d53-477f21a0f677-kube-api-access-cmzk2\" (UniqueName: \"kubernetes.io/projected/371cbd89-b9fb-450b-8d53-477f21a0f677-kube-api-access-cmzk2\") pod \"371cbd89-b9fb-450b-8d53-477f21a0f677\" (UID: \"371cbd89-b9fb-450b-8d53-477f21a0f677\") "
Oct 31 04:56:52.277621 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:52.277549    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-openshift-ca" pod "371cbd89-b9fb-450b-8d53-477f21a0f677" (UID: "371cbd89-b9fb-450b-8d53-477f21a0f677"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:56:52.287527 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:52.287483    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-xks-ca" pod "371cbd89-b9fb-450b-8d53-477f21a0f677" (UID: "371cbd89-b9fb-450b-8d53-477f21a0f677"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:56:52.297792 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:52.297701    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371cbd89-b9fb-450b-8d53-477f21a0f677-kube-api-access-cmzk2" pod "371cbd89-b9fb-450b-8d53-477f21a0f677" (UID: "371cbd89-b9fb-450b-8d53-477f21a0f677"). InnerVolumeSpecName "kube-api-access-cmzk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:56:52.387903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:52.387551    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:52.387903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:52.387603    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371cbd89-b9fb-450b-8d53-477f21a0f677-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:52.387903 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:52.387622    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmzk2\" (UniqueName: \"kubernetes.io/projected/371cbd89-b9fb-450b-8d53-477f21a0f677-kube-api-access-cmzk2\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:56:53.007959 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:52.992187    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b" event={"ID":"371cbd89-b9fb-450b-8d53-477f21a0f677","Type":"ContainerDied","Data":"94dac5aafa4b6f702939a208826e9979d6646138b4a0ba1969086e46fa29238b"}
Oct 31 04:56:53.008284 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:53.008251    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94dac5aafa4b6f702939a208826e9979d6646138b4a0ba1969086e46fa29238b"
Oct 31 04:56:53.008431 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:52.996012    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b040a6ad-jlj8b"
Oct 31 04:56:59.613042 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.613004    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"]
Oct 31 04:56:59.613640 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.613618    5415 cpu_manager.go:414] "containerMap: removing container" podUID="371cbd89-b9fb-450b-8d53-477f21a0f677" containerName="revoke-keys"
Oct 31 04:56:59.613729 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.613642    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371cbd89-b9fb-450b-8d53-477f21a0f677" containerName="revoke-keys"
Oct 31 04:56:59.613788 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.613736    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="371cbd89-b9fb-450b-8d53-477f21a0f677" containerName="revoke-keys"
Oct 31 04:56:59.617262 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.617236    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"
Oct 31 04:56:59.683162 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.683125    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ead1eb-1305-41de-8561-095638754537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d892e0a3-frzpz\" (UID: \"c6ead1eb-1305-41de-8561-095638754537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"
Oct 31 04:56:59.683381 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.683229    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ead1eb-1305-41de-8561-095638754537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d892e0a3-frzpz\" (UID: \"c6ead1eb-1305-41de-8561-095638754537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"
Oct 31 04:56:59.683381 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.683302    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsgkp\" (UniqueName: \"kubernetes.io/projected/c6ead1eb-1305-41de-8561-095638754537-kube-api-access-rsgkp\") pod \"maas-api-revoke-sub-d892e0a3-frzpz\" (UID: \"c6ead1eb-1305-41de-8561-095638754537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"
Oct 31 04:56:59.713599 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.713555    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:56:59.713599 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.713579    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:56:59.715871 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.715846    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"]
Oct 31 04:56:59.783764 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.783720    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsgkp\" (UniqueName: \"kubernetes.io/projected/c6ead1eb-1305-41de-8561-095638754537-kube-api-access-rsgkp\") pod \"maas-api-revoke-sub-d892e0a3-frzpz\" (UID: \"c6ead1eb-1305-41de-8561-095638754537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"
Oct 31 04:56:59.783995 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.783803    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ead1eb-1305-41de-8561-095638754537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d892e0a3-frzpz\" (UID: \"c6ead1eb-1305-41de-8561-095638754537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"
Oct 31 04:56:59.783995 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.783859    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ead1eb-1305-41de-8561-095638754537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d892e0a3-frzpz\" (UID: \"c6ead1eb-1305-41de-8561-095638754537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"
Oct 31 04:56:59.784533 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.784510    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ead1eb-1305-41de-8561-095638754537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d892e0a3-frzpz\" (UID: \"c6ead1eb-1305-41de-8561-095638754537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"
Oct 31 04:56:59.786322 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.786299    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ead1eb-1305-41de-8561-095638754537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d892e0a3-frzpz\" (UID: \"c6ead1eb-1305-41de-8561-095638754537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"
Oct 31 04:56:59.838592 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.838554    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsgkp\" (UniqueName: \"kubernetes.io/projected/c6ead1eb-1305-41de-8561-095638754537-kube-api-access-rsgkp\") pod \"maas-api-revoke-sub-d892e0a3-frzpz\" (UID: \"c6ead1eb-1305-41de-8561-095638754537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"
Oct 31 04:56:59.930154 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:56:59.930064    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"
Oct 31 04:57:00.296282 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:00.296246    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"]
Oct 31 04:57:00.298788 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:57:00.298752    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ead1eb_1305_41de_8561_095638754537.slice/crio-43fbf25e00e67eb37c3352276cfdd39b10cb995af2e4373960ded088a332f4a7 WatchSource:0}: Error finding container 43fbf25e00e67eb37c3352276cfdd39b10cb995af2e4373960ded088a332f4a7: Status 404 returned error can't find the container with id 43fbf25e00e67eb37c3352276cfdd39b10cb995af2e4373960ded088a332f4a7
Oct 31 04:57:01.048131 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:01.048091    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz" event={"ID":"c6ead1eb-1305-41de-8561-095638754537","Type":"ContainerStarted","Data":"345c2710fd95a29b9182e9ed5f0fa97210be90facc25acbbaf2d99ba1540ddee"}
Oct 31 04:57:01.048537 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:01.048157    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz" event={"ID":"c6ead1eb-1305-41de-8561-095638754537","Type":"ContainerStarted","Data":"43fbf25e00e67eb37c3352276cfdd39b10cb995af2e4373960ded088a332f4a7"}
Oct 31 04:57:01.157440 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:01.157384    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz" podStartSLOduration=2.157367109 podStartE2EDuration="2.157367109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:57:01.155108394 +0000 UTC m=+6562.715588144" watchObservedRunningTime="2026-10-31 04:57:01.157367109 +0000 UTC m=+6562.717846839"
Oct 31 04:57:02.057800 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:02.057756    5415 generic.go:356] "Generic (PLEG): container finished" podID="c6ead1eb-1305-41de-8561-095638754537" containerID="345c2710fd95a29b9182e9ed5f0fa97210be90facc25acbbaf2d99ba1540ddee" exitCode=0
Oct 31 04:57:02.058345 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:02.057813    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz" event={"ID":"c6ead1eb-1305-41de-8561-095638754537","Type":"ContainerDied","Data":"345c2710fd95a29b9182e9ed5f0fa97210be90facc25acbbaf2d99ba1540ddee"}
Oct 31 04:57:03.442519 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:03.442489    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"
Oct 31 04:57:03.532789 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:03.532730    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6ead1eb-1305-41de-8561-095638754537-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ead1eb-1305-41de-8561-095638754537-maas-api-xks-ca\") pod \"c6ead1eb-1305-41de-8561-095638754537\" (UID: \"c6ead1eb-1305-41de-8561-095638754537\") "
Oct 31 04:57:03.533014 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:03.532849    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ead1eb-1305-41de-8561-095638754537-kube-api-access-rsgkp\" (UniqueName: \"kubernetes.io/projected/c6ead1eb-1305-41de-8561-095638754537-kube-api-access-rsgkp\") pod \"c6ead1eb-1305-41de-8561-095638754537\" (UID: \"c6ead1eb-1305-41de-8561-095638754537\") "
Oct 31 04:57:03.533014 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:03.532932    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ead1eb-1305-41de-8561-095638754537-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ead1eb-1305-41de-8561-095638754537-maas-api-openshift-ca\") pod \"c6ead1eb-1305-41de-8561-095638754537\" (UID: \"c6ead1eb-1305-41de-8561-095638754537\") "
Oct 31 04:57:03.533544 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:03.533514    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ead1eb-1305-41de-8561-095638754537-maas-api-openshift-ca" pod "c6ead1eb-1305-41de-8561-095638754537" (UID: "c6ead1eb-1305-41de-8561-095638754537"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:57:03.535431 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:03.535406    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ead1eb-1305-41de-8561-095638754537-kube-api-access-rsgkp" pod "c6ead1eb-1305-41de-8561-095638754537" (UID: "c6ead1eb-1305-41de-8561-095638754537"). InnerVolumeSpecName "kube-api-access-rsgkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:57:03.535545 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:03.535521    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ead1eb-1305-41de-8561-095638754537-maas-api-xks-ca" pod "c6ead1eb-1305-41de-8561-095638754537" (UID: "c6ead1eb-1305-41de-8561-095638754537"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:57:03.634231 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:03.634191    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsgkp\" (UniqueName: \"kubernetes.io/projected/c6ead1eb-1305-41de-8561-095638754537-kube-api-access-rsgkp\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:03.634231 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:03.634230    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ead1eb-1305-41de-8561-095638754537-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:03.634452 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:03.634244    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ead1eb-1305-41de-8561-095638754537-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:04.118780 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:04.118748    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz"
Oct 31 04:57:04.119231 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:04.119190    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d892e0a3-frzpz" event={"ID":"c6ead1eb-1305-41de-8561-095638754537","Type":"ContainerDied","Data":"43fbf25e00e67eb37c3352276cfdd39b10cb995af2e4373960ded088a332f4a7"}
Oct 31 04:57:04.119583 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:04.119561    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43fbf25e00e67eb37c3352276cfdd39b10cb995af2e4373960ded088a332f4a7"
Oct 31 04:57:10.327213 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.327170    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"]
Oct 31 04:57:10.327741 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.327582    5415 cpu_manager.go:414] "containerMap: removing container" podUID="c6ead1eb-1305-41de-8561-095638754537" containerName="revoke-keys"
Oct 31 04:57:10.327741 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.327594    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ead1eb-1305-41de-8561-095638754537" containerName="revoke-keys"
Oct 31 04:57:10.327741 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.327672    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ead1eb-1305-41de-8561-095638754537" containerName="revoke-keys"
Oct 31 04:57:10.330899 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.330856    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"
Oct 31 04:57:10.380161 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.380119    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:57:10.401718 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.401680    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94cbe4ce-76wvm\" (UID: \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"
Oct 31 04:57:10.401921 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.401734    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctgq6\" (UniqueName: \"kubernetes.io/projected/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-kube-api-access-ctgq6\") pod \"maas-api-revoke-sub-94cbe4ce-76wvm\" (UID: \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"
Oct 31 04:57:10.401921 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.401826    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94cbe4ce-76wvm\" (UID: \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"
Oct 31 04:57:10.427340 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.427310    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:57:10.429414 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.429385    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"]
Oct 31 04:57:10.503136 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.503097    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94cbe4ce-76wvm\" (UID: \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"
Oct 31 04:57:10.503330 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.503161    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctgq6\" (UniqueName: \"kubernetes.io/projected/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-kube-api-access-ctgq6\") pod \"maas-api-revoke-sub-94cbe4ce-76wvm\" (UID: \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"
Oct 31 04:57:10.503330 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.503228    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94cbe4ce-76wvm\" (UID: \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"
Oct 31 04:57:10.503963 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.503944    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94cbe4ce-76wvm\" (UID: \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"
Oct 31 04:57:10.505598 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.505572    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94cbe4ce-76wvm\" (UID: \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"
Oct 31 04:57:10.557344 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.557305    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctgq6\" (UniqueName: \"kubernetes.io/projected/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-kube-api-access-ctgq6\") pod \"maas-api-revoke-sub-94cbe4ce-76wvm\" (UID: \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"
Oct 31 04:57:10.644841 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:10.644803    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"
Oct 31 04:57:11.011668 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:11.011644    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"]
Oct 31 04:57:11.013287 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:57:11.013256    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae5bc24_c2b4_436f_84b9_d1e5dd684fa1.slice/crio-1076100d255b6d61e397e95a92324fd0f68520fe9e5a89e7f01e19698cf2f000 WatchSource:0}: Error finding container 1076100d255b6d61e397e95a92324fd0f68520fe9e5a89e7f01e19698cf2f000: Status 404 returned error can't find the container with id 1076100d255b6d61e397e95a92324fd0f68520fe9e5a89e7f01e19698cf2f000
Oct 31 04:57:11.166781 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:11.166740    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm" event={"ID":"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1","Type":"ContainerStarted","Data":"1076100d255b6d61e397e95a92324fd0f68520fe9e5a89e7f01e19698cf2f000"}
Oct 31 04:57:12.173571 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:12.173535    5415 generic.go:356] "Generic (PLEG): container finished" podID="2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1" containerID="c5583682188a3b4d59b8fbcabf989eb3ba8ec8a2bf47fdb95a30910d73d08e2b" exitCode=0
Oct 31 04:57:12.173993 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:12.173624    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm" event={"ID":"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1","Type":"ContainerDied","Data":"c5583682188a3b4d59b8fbcabf989eb3ba8ec8a2bf47fdb95a30910d73d08e2b"}
Oct 31 04:57:13.350752 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:13.350711    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"
Oct 31 04:57:13.434964 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:13.434842    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-kube-api-access-ctgq6\" (UniqueName: \"kubernetes.io/projected/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-kube-api-access-ctgq6\") pod \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\" (UID: \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\") "
Oct 31 04:57:13.434964 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:13.434926    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-xks-ca\") pod \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\" (UID: \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\") "
Oct 31 04:57:13.435226 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:13.434995    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-openshift-ca\") pod \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\" (UID: \"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1\") "
Oct 31 04:57:13.435591 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:13.435550    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-openshift-ca" pod "2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1" (UID: "2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:57:13.437735 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:13.437700    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-xks-ca" pod "2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1" (UID: "2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:57:13.437843 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:13.437745    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-kube-api-access-ctgq6" pod "2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1" (UID: "2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1"). InnerVolumeSpecName "kube-api-access-ctgq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:57:13.536479 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:13.536428    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:13.536479 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:13.536478    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctgq6\" (UniqueName: \"kubernetes.io/projected/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-kube-api-access-ctgq6\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:13.536772 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:13.536495    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:14.208912 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:14.201825    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm" event={"ID":"2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1","Type":"ContainerDied","Data":"1076100d255b6d61e397e95a92324fd0f68520fe9e5a89e7f01e19698cf2f000"}
Oct 31 04:57:14.209257 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:14.209227    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1076100d255b6d61e397e95a92324fd0f68520fe9e5a89e7f01e19698cf2f000"
Oct 31 04:57:14.209388 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:14.206034    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbe4ce-76wvm"
Oct 31 04:57:16.174374 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:57:16.174298    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:57:20.042718 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:20.042681    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"]
Oct 31 04:57:20.089446 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:20.089398    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fb7551-sj26p"]
Oct 31 04:57:20.604724 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:20.604687    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68271017-34eb-41ed-a4c9-d9d3db251d7f" path="/var/lib/kubelet/pods/68271017-34eb-41ed-a4c9-d9d3db251d7f/volumes"
Oct 31 04:57:21.081849 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.081761    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"]
Oct 31 04:57:21.082249 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.082185    5415 cpu_manager.go:414] "containerMap: removing container" podUID="2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1" containerName="revoke-keys"
Oct 31 04:57:21.082249 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.082199    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1" containerName="revoke-keys"
Oct 31 04:57:21.082372 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.082291    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1" containerName="revoke-keys"
Oct 31 04:57:21.085593 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.085570    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"
Oct 31 04:57:21.135495 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.135467    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:57:21.182414 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.182380    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:57:21.184624 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.184602    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"]
Oct 31 04:57:21.234710 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.234673    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5cb998e-lrqbt\" (UID: \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"
Oct 31 04:57:21.234910 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.234731    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlw7j\" (UniqueName: \"kubernetes.io/projected/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-kube-api-access-hlw7j\") pod \"maas-api-revoke-sub-b5cb998e-lrqbt\" (UID: \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"
Oct 31 04:57:21.234910 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.234895    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5cb998e-lrqbt\" (UID: \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"
Oct 31 04:57:21.335565 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.335534    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw7j\" (UniqueName: \"kubernetes.io/projected/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-kube-api-access-hlw7j\") pod \"maas-api-revoke-sub-b5cb998e-lrqbt\" (UID: \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"
Oct 31 04:57:21.335765 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.335612    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5cb998e-lrqbt\" (UID: \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"
Oct 31 04:57:21.335765 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.335660    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5cb998e-lrqbt\" (UID: \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"
Oct 31 04:57:21.336307 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.336285    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5cb998e-lrqbt\" (UID: \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"
Oct 31 04:57:21.338207 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.338177    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5cb998e-lrqbt\" (UID: \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"
Oct 31 04:57:21.390489 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.390451    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlw7j\" (UniqueName: \"kubernetes.io/projected/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-kube-api-access-hlw7j\") pod \"maas-api-revoke-sub-b5cb998e-lrqbt\" (UID: \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"
Oct 31 04:57:21.398599 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.398573    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"
Oct 31 04:57:21.766269 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:21.766245    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"]
Oct 31 04:57:21.768589 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:57:21.768561    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1a30a5_e0b3_4ccc_92be_5c2e0a7de291.slice/crio-41945df165aa7f64ed7cdf08c7314554c7bc7a8d241a253479334c9e8674208c WatchSource:0}: Error finding container 41945df165aa7f64ed7cdf08c7314554c7bc7a8d241a253479334c9e8674208c: Status 404 returned error can't find the container with id 41945df165aa7f64ed7cdf08c7314554c7bc7a8d241a253479334c9e8674208c
Oct 31 04:57:22.253924 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:22.253868    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt" event={"ID":"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291","Type":"ContainerStarted","Data":"dacfaaa433761aaf300516360722326922e181e6febf1c19963bc227b031aabf"}
Oct 31 04:57:22.254311 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:22.253952    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt" event={"ID":"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291","Type":"ContainerStarted","Data":"41945df165aa7f64ed7cdf08c7314554c7bc7a8d241a253479334c9e8674208c"}
Oct 31 04:57:22.362548 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:22.362488    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt" podStartSLOduration=1.362472675 podStartE2EDuration="1.362472675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:57:22.361417213 +0000 UTC m=+6583.921896944" watchObservedRunningTime="2026-10-31 04:57:22.362472675 +0000 UTC m=+6583.922952406"
Oct 31 04:57:23.261052 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:23.261016    5415 generic.go:356] "Generic (PLEG): container finished" podID="4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291" containerID="dacfaaa433761aaf300516360722326922e181e6febf1c19963bc227b031aabf" exitCode=0
Oct 31 04:57:23.261489 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:23.261110    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt" event={"ID":"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291","Type":"ContainerDied","Data":"dacfaaa433761aaf300516360722326922e181e6febf1c19963bc227b031aabf"}
Oct 31 04:57:24.646964 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:24.646931    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"
Oct 31 04:57:24.734446 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:24.734404    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-openshift-ca\") pod \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\" (UID: \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\") "
Oct 31 04:57:24.734446 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:24.734448    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-kube-api-access-hlw7j\" (UniqueName: \"kubernetes.io/projected/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-kube-api-access-hlw7j\") pod \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\" (UID: \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\") "
Oct 31 04:57:24.734661 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:24.734499    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-xks-ca\") pod \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\" (UID: \"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291\") "
Oct 31 04:57:24.735143 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:24.735113    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-openshift-ca" pod "4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291" (UID: "4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:57:24.736704 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:24.736669    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-kube-api-access-hlw7j" pod "4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291" (UID: "4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291"). InnerVolumeSpecName "kube-api-access-hlw7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:57:24.736812 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:24.736724    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-xks-ca" pod "4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291" (UID: "4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:57:24.835151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:24.835114    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:24.835151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:24.835141    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:24.835151 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:24.835152    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlw7j\" (UniqueName: \"kubernetes.io/projected/4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291-kube-api-access-hlw7j\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:25.309901 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:25.309708    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt" event={"ID":"4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291","Type":"ContainerDied","Data":"41945df165aa7f64ed7cdf08c7314554c7bc7a8d241a253479334c9e8674208c"}
Oct 31 04:57:25.309901 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:25.309779    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41945df165aa7f64ed7cdf08c7314554c7bc7a8d241a253479334c9e8674208c"
Oct 31 04:57:25.309901 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:25.309859    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cb998e-lrqbt"
Oct 31 04:57:31.044022 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.043985    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"]
Oct 31 04:57:31.090750 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.090714    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8affa336-gkdkp"]
Oct 31 04:57:31.804649 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.804610    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"]
Oct 31 04:57:31.805034 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.805020    5415 cpu_manager.go:414] "containerMap: removing container" podUID="4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291" containerName="revoke-keys"
Oct 31 04:57:31.805086 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.805035    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291" containerName="revoke-keys"
Oct 31 04:57:31.805133 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.805124    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291" containerName="revoke-keys"
Oct 31 04:57:31.808253 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.808229    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"
Oct 31 04:57:31.858018 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.857984    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:57:31.890521 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.890484    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bc3392e-gvzbt\" (UID: \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"
Oct 31 04:57:31.890730 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.890526    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bc3392e-gvzbt\" (UID: \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"
Oct 31 04:57:31.890730 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.890672    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpv7l\" (UniqueName: \"kubernetes.io/projected/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-kube-api-access-wpv7l\") pod \"maas-api-revoke-sub-5bc3392e-gvzbt\" (UID: \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"
Oct 31 04:57:31.904783 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.904753    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:57:31.906957 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.906933    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"]
Oct 31 04:57:31.991957 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.991920    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bc3392e-gvzbt\" (UID: \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"
Oct 31 04:57:31.992181 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.991968    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bc3392e-gvzbt\" (UID: \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"
Oct 31 04:57:31.992181 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.992013    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpv7l\" (UniqueName: \"kubernetes.io/projected/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-kube-api-access-wpv7l\") pod \"maas-api-revoke-sub-5bc3392e-gvzbt\" (UID: \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"
Oct 31 04:57:31.992734 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.992708    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bc3392e-gvzbt\" (UID: \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"
Oct 31 04:57:31.994443 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:31.994422    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bc3392e-gvzbt\" (UID: \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"
Oct 31 04:57:32.046403 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:32.046369    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpv7l\" (UniqueName: \"kubernetes.io/projected/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-kube-api-access-wpv7l\") pod \"maas-api-revoke-sub-5bc3392e-gvzbt\" (UID: \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"
Oct 31 04:57:32.121385 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:32.121338    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"
Oct 31 04:57:32.490550 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:32.490528    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"]
Oct 31 04:57:32.493128 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:57:32.493096    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cee1c6_1c47_44ce_8dc9_0ce7e373ed43.slice/crio-4dfb8eb5da3bcbde7a6065dc98e167ad9a78cfcb29a85506d404a2747fe43e52 WatchSource:0}: Error finding container 4dfb8eb5da3bcbde7a6065dc98e167ad9a78cfcb29a85506d404a2747fe43e52: Status 404 returned error can't find the container with id 4dfb8eb5da3bcbde7a6065dc98e167ad9a78cfcb29a85506d404a2747fe43e52
Oct 31 04:57:32.604273 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:32.604242    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d547054-9913-4eab-bb46-5d707acf471c" path="/var/lib/kubelet/pods/9d547054-9913-4eab-bb46-5d707acf471c/volumes"
Oct 31 04:57:33.355170 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:33.355132    5415 generic.go:356] "Generic (PLEG): container finished" podID="e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43" containerID="499db9fa4dd5de6bdcff29e725827e83dc76ea96a15a656e50f657cd2f2f2f0a" exitCode=0
Oct 31 04:57:33.355574 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:33.355219    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt" event={"ID":"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43","Type":"ContainerDied","Data":"499db9fa4dd5de6bdcff29e725827e83dc76ea96a15a656e50f657cd2f2f2f0a"}
Oct 31 04:57:33.355574 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:33.355269    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt" event={"ID":"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43","Type":"ContainerStarted","Data":"4dfb8eb5da3bcbde7a6065dc98e167ad9a78cfcb29a85506d404a2747fe43e52"}
Oct 31 04:57:34.550942 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:34.550912    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"
Oct 31 04:57:34.613623 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:34.613536    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-openshift-ca\") pod \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\" (UID: \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\") "
Oct 31 04:57:34.613799 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:34.613638    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-xks-ca\") pod \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\" (UID: \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\") "
Oct 31 04:57:34.613799 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:34.613741    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-kube-api-access-wpv7l\" (UniqueName: \"kubernetes.io/projected/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-kube-api-access-wpv7l\") pod \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\" (UID: \"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43\") "
Oct 31 04:57:34.614025 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:34.613993    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-openshift-ca" pod "e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43" (UID: "e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:57:34.615870 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:34.615841    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-kube-api-access-wpv7l" pod "e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43" (UID: "e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43"). InnerVolumeSpecName "kube-api-access-wpv7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:57:34.616325 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:34.616300    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-xks-ca" pod "e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43" (UID: "e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:57:34.715641 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:34.715600    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:34.715641 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:34.715643    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpv7l\" (UniqueName: \"kubernetes.io/projected/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-kube-api-access-wpv7l\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:34.715915 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:34.715659    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:35.393902 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:35.393817    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt" event={"ID":"e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43","Type":"ContainerDied","Data":"4dfb8eb5da3bcbde7a6065dc98e167ad9a78cfcb29a85506d404a2747fe43e52"}
Oct 31 04:57:35.394135 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:35.393931    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dfb8eb5da3bcbde7a6065dc98e167ad9a78cfcb29a85506d404a2747fe43e52"
Oct 31 04:57:35.394135 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:35.394013    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc3392e-gvzbt"
Oct 31 04:57:42.607967 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.607935    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"]
Oct 31 04:57:42.608497 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.608350    5415 cpu_manager.go:414] "containerMap: removing container" podUID="e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43" containerName="revoke-keys"
Oct 31 04:57:42.608497 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.608361    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43" containerName="revoke-keys"
Oct 31 04:57:42.608497 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.608436    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43" containerName="revoke-keys"
Oct 31 04:57:42.613012 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.612983    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"
Oct 31 04:57:42.664147 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.664114    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:57:42.700956 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.700919    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lbr9\" (UniqueName: \"kubernetes.io/projected/1a821a56-7ace-42f8-8edb-9a4c9374ad41-kube-api-access-8lbr9\") pod \"maas-api-revoke-sub-bc9c3ab3-77m9h\" (UID: \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"
Oct 31 04:57:42.701163 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.700969    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9c3ab3-77m9h\" (UID: \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"
Oct 31 04:57:42.701163 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.701116    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9c3ab3-77m9h\" (UID: \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"
Oct 31 04:57:42.710958 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.710928    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:57:42.713249 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.713227    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"]
Oct 31 04:57:42.801625 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.801582    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9c3ab3-77m9h\" (UID: \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"
Oct 31 04:57:42.801848 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.801677    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lbr9\" (UniqueName: \"kubernetes.io/projected/1a821a56-7ace-42f8-8edb-9a4c9374ad41-kube-api-access-8lbr9\") pod \"maas-api-revoke-sub-bc9c3ab3-77m9h\" (UID: \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"
Oct 31 04:57:42.801848 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.801697    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9c3ab3-77m9h\" (UID: \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"
Oct 31 04:57:42.802343 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.802315    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9c3ab3-77m9h\" (UID: \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"
Oct 31 04:57:42.804157 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.804134    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9c3ab3-77m9h\" (UID: \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"
Oct 31 04:57:42.855670 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.855635    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lbr9\" (UniqueName: \"kubernetes.io/projected/1a821a56-7ace-42f8-8edb-9a4c9374ad41-kube-api-access-8lbr9\") pod \"maas-api-revoke-sub-bc9c3ab3-77m9h\" (UID: \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"
Oct 31 04:57:42.926668 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:42.926571    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"
Oct 31 04:57:43.294119 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:43.294094    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"]
Oct 31 04:57:43.296710 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:57:43.296678    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a821a56_7ace_42f8_8edb_9a4c9374ad41.slice/crio-71c46bdd96429118ae6bc9e3e7ef27a3da60ca9d544b5b2641ebbfb37e2b3934 WatchSource:0}: Error finding container 71c46bdd96429118ae6bc9e3e7ef27a3da60ca9d544b5b2641ebbfb37e2b3934: Status 404 returned error can't find the container with id 71c46bdd96429118ae6bc9e3e7ef27a3da60ca9d544b5b2641ebbfb37e2b3934
Oct 31 04:57:43.455799 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:43.455764    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h" event={"ID":"1a821a56-7ace-42f8-8edb-9a4c9374ad41","Type":"ContainerStarted","Data":"71c46bdd96429118ae6bc9e3e7ef27a3da60ca9d544b5b2641ebbfb37e2b3934"}
Oct 31 04:57:44.463004 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:44.462917    5415 generic.go:356] "Generic (PLEG): container finished" podID="1a821a56-7ace-42f8-8edb-9a4c9374ad41" containerID="6e331e510813e0a6f311e58de63a6dd1c5e1afaa74977f675dd1923cd8720a3b" exitCode=0
Oct 31 04:57:44.463381 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:44.463002    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h" event={"ID":"1a821a56-7ace-42f8-8edb-9a4c9374ad41","Type":"ContainerDied","Data":"6e331e510813e0a6f311e58de63a6dd1c5e1afaa74977f675dd1923cd8720a3b"}
Oct 31 04:57:45.795738 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:45.795708    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"
Oct 31 04:57:45.842590 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:45.842542    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a821a56-7ace-42f8-8edb-9a4c9374ad41-kube-api-access-8lbr9\" (UniqueName: \"kubernetes.io/projected/1a821a56-7ace-42f8-8edb-9a4c9374ad41-kube-api-access-8lbr9\") pod \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\" (UID: \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\") "
Oct 31 04:57:45.842784 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:45.842649    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-xks-ca\") pod \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\" (UID: \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\") "
Oct 31 04:57:45.842784 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:45.842690    5415 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-openshift-ca\") pod \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\" (UID: \"1a821a56-7ace-42f8-8edb-9a4c9374ad41\") "
Oct 31 04:57:45.843121 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:45.843095    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-openshift-ca" pod "1a821a56-7ace-42f8-8edb-9a4c9374ad41" (UID: "1a821a56-7ace-42f8-8edb-9a4c9374ad41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:57:45.844705 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:45.844669    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-xks-ca" pod "1a821a56-7ace-42f8-8edb-9a4c9374ad41" (UID: "1a821a56-7ace-42f8-8edb-9a4c9374ad41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:57:45.844814 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:45.844732    5415 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a821a56-7ace-42f8-8edb-9a4c9374ad41-kube-api-access-8lbr9" pod "1a821a56-7ace-42f8-8edb-9a4c9374ad41" (UID: "1a821a56-7ace-42f8-8edb-9a4c9374ad41"). InnerVolumeSpecName "kube-api-access-8lbr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:57:45.943348 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:45.943306    5415 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lbr9\" (UniqueName: \"kubernetes.io/projected/1a821a56-7ace-42f8-8edb-9a4c9374ad41-kube-api-access-8lbr9\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:45.943348 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:45.943342    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-xks-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:45.943348 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:45.943355    5415 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a821a56-7ace-42f8-8edb-9a4c9374ad41-maas-api-openshift-ca\") on node \"ip-10-0-139-164.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:57:46.235527 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:57:46.235481    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:57:46.515553 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:46.515404    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h" event={"ID":"1a821a56-7ace-42f8-8edb-9a4c9374ad41","Type":"ContainerDied","Data":"71c46bdd96429118ae6bc9e3e7ef27a3da60ca9d544b5b2641ebbfb37e2b3934"}
Oct 31 04:57:46.515843 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:46.515794    5415 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71c46bdd96429118ae6bc9e3e7ef27a3da60ca9d544b5b2641ebbfb37e2b3934"
Oct 31 04:57:46.516143 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:57:46.516071    5415 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9c3ab3-77m9h"
Oct 31 04:58:07.049899 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:07.049796    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"]
Oct 31 04:58:07.096654 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:07.096617    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b907400c-8lwps"]
Oct 31 04:58:08.605088 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:08.605049    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14ab89e-f2d6-4615-84bd-1a380d2440c9" path="/var/lib/kubelet/pods/e14ab89e-f2d6-4615-84bd-1a380d2440c9/volumes"
Oct 31 04:58:16.174933 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:58:16.174890    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:58:27.366745 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:27.366710    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qwnr5_81594aa6-a125-4f16-9c62-5b3b6f24c6fb/manager/0.log"
Oct 31 04:58:27.867153 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:27.867120    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cmm9t_e94e222c-4de2-4046-a659-0c92420cc03e/manager/0.log"
Oct 31 04:58:28.515291 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:28.515260    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4m5hw_35e1e5d7-2240-4292-9f78-a9676f67e28a/server/0.log"
Oct 31 04:58:28.735475 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:28.735432    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hpjht_6cb52f07-5218-4b28-896c-9b97e062ef44/manager/0.log"
Oct 31 04:58:28.984203 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:28.984171    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-bk6x5_9dcba361-a1d8-4e62-843c-be420bd3fea8/proxy/0.log"
Oct 31 04:58:30.436639 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:30.436608    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-04dc118f-6ds4t_1d5a92ba-20bc-40e6-986a-d0449d611210/revoke-keys/0.log"
Oct 31 04:58:30.655979 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:30.655950    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0e5c068e-98rkk_a1a1e27a-db48-4b9d-8775-f0751efb35f6/revoke-keys/0.log"
Oct 31 04:58:30.881285 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:30.881254    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1c89297c-ztfr7_70d3e47e-18e6-4255-8fa4-824e7ed524d1/revoke-keys/0.log"
Oct 31 04:58:31.111570 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:31.111543    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4d8262f3-v4tvc_5611f194-dbb2-47bd-b913-3ffc8ac4ea92/revoke-keys/0.log"
Oct 31 04:58:31.322857 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:31.322770    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-52fd21f9-krtfh_a27d5840-4f8e-46e6-82b5-41c09686e1c6/revoke-keys/0.log"
Oct 31 04:58:31.536343 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:31.536301    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5bc3392e-gvzbt_e9cee1c6-1c47-44ce-8dc9-0ce7e373ed43/revoke-keys/0.log"
Oct 31 04:58:31.758193 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:31.758164    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8833e2d9-qjblt_35113590-55f3-4a69-a0bf-0d107fcf52b3/revoke-keys/0.log"
Oct 31 04:58:31.978056 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:31.978024    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-94cbe4ce-76wvm_2ae5bc24-c2b4-436f-84b9-d1e5dd684fa1/revoke-keys/0.log"
Oct 31 04:58:32.179465 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:32.179435    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a94ae629-xzgjb_e5cfac24-eb41-424d-94a1-4bc541bf90a2/revoke-keys/0.log"
Oct 31 04:58:32.400192 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:32.400160    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b040a6ad-jlj8b_371cbd89-b9fb-450b-8d53-477f21a0f677/revoke-keys/0.log"
Oct 31 04:58:32.611564 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:32.611526    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b3fa8702-qff4w_9ded1b2e-cb23-403a-bd07-3dcfaa0bea95/revoke-keys/0.log"
Oct 31 04:58:32.832114 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:32.832077    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b5cb998e-lrqbt_4f1a30a5-e0b3-4ccc-92be-5c2e0a7de291/revoke-keys/0.log"
Oct 31 04:58:33.054018 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:33.053938    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bc9c3ab3-77m9h_1a821a56-7ace-42f8-8edb-9a4c9374ad41/revoke-keys/0.log"
Oct 31 04:58:33.277128 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:33.277100    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d892e0a3-frzpz_c6ead1eb-1305-41de-8561-095638754537/revoke-keys/0.log"
Oct 31 04:58:33.485820 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:33.485793    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-efb1238d-45clm_4f7d679c-f688-4470-9d8d-71f27139997d/revoke-keys/0.log"
Oct 31 04:58:33.696582 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:33.696551    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ff6aa3d8-wt5pw_2bd0bcdf-26a5-4045-826b-54e5e77dc3a1/revoke-keys/0.log"
Oct 31 04:58:33.917965 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:33.917937    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-2hpz5_c934f25e-6d90-4da4-9594-009240e41070/postgres/0.log"
Oct 31 04:58:36.768773 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:36.768745    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-f4q4d_b4e3fc57-e3fd-4a86-954c-70cf7062810f/kuadrant-console-plugin/0.log"
Oct 31 04:58:37.629965 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:37.629930    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nmdgv_6e19afe0-f05b-4294-8d01-8d04cbb5b833/manager/0.log"
Oct 31 04:58:39.281661 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:39.281630    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb6b5fd89-w7s8t_5ec9b273-54b7-49b5-bd15-09d2230e7c3c/kube-auth-proxy/0.log"
Oct 31 04:58:39.497748 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:39.497708    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-5smg5_1cfc6a64-edfa-4e51-83b5-f38f438f1877/istio-proxy/0.log"
Oct 31 04:58:40.197864 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:40.197832    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-bwzrc_c9d3bfd0-545c-4be0-92ae-30a7c68138ca/istio-proxy/0.log"
Oct 31 04:58:40.215662 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:40.215622    5415 scope.go:122] "RemoveContainer" containerID="ab4c9ce7dfdcdc5dfe8ec49d93b510473c73bada5310aa5371244d5137b06a09"
Oct 31 04:58:40.224991 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:40.224958    5415 scope.go:122] "RemoveContainer" containerID="930f50174697502360e9bd147744d33e495d6f63e05780d5f891c5e1014d057a"
Oct 31 04:58:41.592803 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:41.592773    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-wlt76_c0469dfd-14dc-458a-9b3e-d5b0ecc25000/istio-proxy/0.log"
Oct 31 04:58:46.175152 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:58:46.175064    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:58:49.043105 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:49.043063    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"]
Oct 31 04:58:49.089916 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:49.089866    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04dc118f-6ds4t"]
Oct 31 04:58:50.604444 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:50.604409    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d5a92ba-20bc-40e6-986a-d0449d611210" path="/var/lib/kubelet/pods/1d5a92ba-20bc-40e6-986a-d0449d611210/volumes"
Oct 31 04:58:55.435278 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.435244    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m7rwt/must-gather-llqkx"]
Oct 31 04:58:55.435723 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.435706    5415 cpu_manager.go:414] "containerMap: removing container" podUID="1a821a56-7ace-42f8-8edb-9a4c9374ad41" containerName="revoke-keys"
Oct 31 04:58:55.435787 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.435724    5415 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a821a56-7ace-42f8-8edb-9a4c9374ad41" containerName="revoke-keys"
Oct 31 04:58:55.435838 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.435791    5415 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a821a56-7ace-42f8-8edb-9a4c9374ad41" containerName="revoke-keys"
Oct 31 04:58:55.440740 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.440718    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m7rwt/must-gather-llqkx"
Oct 31 04:58:55.474183 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.474148    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mllpx\" (UniqueName: \"kubernetes.io/projected/7768ecc5-73e7-48af-8dc5-5d207a36802e-kube-api-access-mllpx\") pod \"must-gather-llqkx\" (UID: \"7768ecc5-73e7-48af-8dc5-5d207a36802e\") " pod="openshift-must-gather-m7rwt/must-gather-llqkx"
Oct 31 04:58:55.474365 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.474209    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7768ecc5-73e7-48af-8dc5-5d207a36802e-must-gather-output\") pod \"must-gather-llqkx\" (UID: \"7768ecc5-73e7-48af-8dc5-5d207a36802e\") " pod="openshift-must-gather-m7rwt/must-gather-llqkx"
Oct 31 04:58:55.495219 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.495184    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m7rwt/must-gather-llqkx"]
Oct 31 04:58:55.540057 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.540016    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m7rwt\"/\"kube-root-ca.crt\""
Oct 31 04:58:55.540057 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.540037    5415 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m7rwt\"/\"default-dockercfg-74bbv\""
Oct 31 04:58:55.540057 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.540016    5415 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m7rwt\"/\"openshift-service-ca.crt\""
Oct 31 04:58:55.575380 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.575338    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mllpx\" (UniqueName: \"kubernetes.io/projected/7768ecc5-73e7-48af-8dc5-5d207a36802e-kube-api-access-mllpx\") pod \"must-gather-llqkx\" (UID: \"7768ecc5-73e7-48af-8dc5-5d207a36802e\") " pod="openshift-must-gather-m7rwt/must-gather-llqkx"
Oct 31 04:58:55.575582 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.575399    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7768ecc5-73e7-48af-8dc5-5d207a36802e-must-gather-output\") pod \"must-gather-llqkx\" (UID: \"7768ecc5-73e7-48af-8dc5-5d207a36802e\") " pod="openshift-must-gather-m7rwt/must-gather-llqkx"
Oct 31 04:58:55.575772 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.575753    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7768ecc5-73e7-48af-8dc5-5d207a36802e-must-gather-output\") pod \"must-gather-llqkx\" (UID: \"7768ecc5-73e7-48af-8dc5-5d207a36802e\") " pod="openshift-must-gather-m7rwt/must-gather-llqkx"
Oct 31 04:58:55.629035 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.629000    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mllpx\" (UniqueName: \"kubernetes.io/projected/7768ecc5-73e7-48af-8dc5-5d207a36802e-kube-api-access-mllpx\") pod \"must-gather-llqkx\" (UID: \"7768ecc5-73e7-48af-8dc5-5d207a36802e\") " pod="openshift-must-gather-m7rwt/must-gather-llqkx"
Oct 31 04:58:55.755215 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:55.755123    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m7rwt/must-gather-llqkx"
Oct 31 04:58:56.119907 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:56.119861    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m7rwt/must-gather-llqkx"]
Oct 31 04:58:56.121954 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:58:56.121921    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7768ecc5_73e7_48af_8dc5_5d207a36802e.slice/crio-eefc86450ea19fd39149ecb5ba8fc41d7ed34ebe6341ba0b0350ed09c5da7a93 WatchSource:0}: Error finding container eefc86450ea19fd39149ecb5ba8fc41d7ed34ebe6341ba0b0350ed09c5da7a93: Status 404 returned error can't find the container with id eefc86450ea19fd39149ecb5ba8fc41d7ed34ebe6341ba0b0350ed09c5da7a93
Oct 31 04:58:56.123943 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:56.123919    5415 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:58:56.914022 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:56.913979    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m7rwt/must-gather-llqkx" event={"ID":"7768ecc5-73e7-48af-8dc5-5d207a36802e","Type":"ContainerStarted","Data":"eefc86450ea19fd39149ecb5ba8fc41d7ed34ebe6341ba0b0350ed09c5da7a93"}
Oct 31 04:58:57.921214 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:57.921173    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m7rwt/must-gather-llqkx" event={"ID":"7768ecc5-73e7-48af-8dc5-5d207a36802e","Type":"ContainerStarted","Data":"c2a3bf950651c13675ab1d108fa7db9c2f1523c42628fb49a8853999189e45cc"}
Oct 31 04:58:57.921655 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:57.921226    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m7rwt/must-gather-llqkx" event={"ID":"7768ecc5-73e7-48af-8dc5-5d207a36802e","Type":"ContainerStarted","Data":"eed9bc9684933d0acaacbcb3d39b86b019f98b5228f2936f33bc55ade7000089"}
Oct 31 04:58:58.028897 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:58:58.028827    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-m7rwt/must-gather-llqkx" podStartSLOduration=1.623716832 podStartE2EDuration="3.028810939s" totalImagesPullingTime="1.405094107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:58:58.027502951 +0000 UTC m=+6679.587982679" watchObservedRunningTime="2026-10-31 04:58:58.028810939 +0000 UTC m=+6679.589290713"
Oct 31 04:59:00.042826 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:00.042789    5415 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"]
Oct 31 04:59:00.088663 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:00.088624    5415 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c89297c-ztfr7"]
Oct 31 04:59:00.604429 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:00.604397    5415 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70d3e47e-18e6-4255-8fa4-824e7ed524d1" path="/var/lib/kubelet/pods/70d3e47e-18e6-4255-8fa4-824e7ed524d1/volumes"
Oct 31 04:59:04.526899 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:04.526535    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ldcbk_d35c8150-ce21-4660-ad5d-e5cc1fce684b/global-pull-secret-syncer/0.log"
Oct 31 04:59:07.769899 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:07.769259    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x4wzm_17bd527b-9dd9-4fc6-86b3-57b5456c5a6b/konnectivity-agent/0.log"
Oct 31 04:59:07.919317 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:07.919274    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qwnr5_81594aa6-a125-4f16-9c62-5b3b6f24c6fb/manager/0.log"
Oct 31 04:59:08.055126 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:08.055040    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qwnr5_81594aa6-a125-4f16-9c62-5b3b6f24c6fb/manager/0.log"
Oct 31 04:59:08.560322 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:08.560284    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-164.us-west-2.compute.internal_d510a50aad744fb4754386c4136c4a71/haproxy/0.log"
Oct 31 04:59:09.142209 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:09.142140    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cmm9t_e94e222c-4de2-4046-a659-0c92420cc03e/manager/0.log"
Oct 31 04:59:09.314612 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:09.312539    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cmm9t_e94e222c-4de2-4046-a659-0c92420cc03e/manager/0.log"
Oct 31 04:59:09.484256 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:09.484184    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4m5hw_35e1e5d7-2240-4292-9f78-a9676f67e28a/server/0.log"
Oct 31 04:59:09.691898 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:09.691780    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4m5hw_35e1e5d7-2240-4292-9f78-a9676f67e28a/server/0.log"
Oct 31 04:59:12.188688 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:12.188646    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hpjht_6cb52f07-5218-4b28-896c-9b97e062ef44/manager/0.log"
Oct 31 04:59:12.333611 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:12.332315    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-bk6x5_9dcba361-a1d8-4e62-843c-be420bd3fea8/proxy/0.log"
Oct 31 04:59:12.385362 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:12.385329    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hpjht_6cb52f07-5218-4b28-896c-9b97e062ef44/manager/0.log"
Oct 31 04:59:12.506294 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:12.506215    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-bk6x5_9dcba361-a1d8-4e62-843c-be420bd3fea8/proxy/0.log"
Oct 31 04:59:14.917684 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:14.917653    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-f4q4d_b4e3fc57-e3fd-4a86-954c-70cf7062810f/kuadrant-console-plugin/0.log"
Oct 31 04:59:15.164366 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.164333    5415 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"]
Oct 31 04:59:15.170227 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.170159    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.226365 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.226330    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"]
Oct 31 04:59:15.262010 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.261978    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/244ce30f-9471-4bcc-a3ef-9e398eb40585-podres\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.262213 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.262024    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/244ce30f-9471-4bcc-a3ef-9e398eb40585-sys\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.262213 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.262165    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lnj8\" (UniqueName: \"kubernetes.io/projected/244ce30f-9471-4bcc-a3ef-9e398eb40585-kube-api-access-8lnj8\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.262308 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.262220    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/244ce30f-9471-4bcc-a3ef-9e398eb40585-lib-modules\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.262343 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.262320    5415 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/244ce30f-9471-4bcc-a3ef-9e398eb40585-proc\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.363371 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.363330    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/244ce30f-9471-4bcc-a3ef-9e398eb40585-podres\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.363371 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.363379    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/244ce30f-9471-4bcc-a3ef-9e398eb40585-sys\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.363611 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.363416    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lnj8\" (UniqueName: \"kubernetes.io/projected/244ce30f-9471-4bcc-a3ef-9e398eb40585-kube-api-access-8lnj8\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.363611 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.363441    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/244ce30f-9471-4bcc-a3ef-9e398eb40585-lib-modules\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.363611 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.363475    5415 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/244ce30f-9471-4bcc-a3ef-9e398eb40585-proc\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.363611 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.363472    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/244ce30f-9471-4bcc-a3ef-9e398eb40585-podres\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.363611 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.363525    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/244ce30f-9471-4bcc-a3ef-9e398eb40585-sys\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.363611 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.363576    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/244ce30f-9471-4bcc-a3ef-9e398eb40585-lib-modules\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.363611 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.363587    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/244ce30f-9471-4bcc-a3ef-9e398eb40585-proc\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.375722 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.375698    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qwnr5_81594aa6-a125-4f16-9c62-5b3b6f24c6fb/manager/0.log"
Oct 31 04:59:15.418109 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.418072    5415 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lnj8\" (UniqueName: \"kubernetes.io/projected/244ce30f-9471-4bcc-a3ef-9e398eb40585-kube-api-access-8lnj8\") pod \"perf-node-gather-daemonset-hs648\" (UID: \"244ce30f-9471-4bcc-a3ef-9e398eb40585\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.482607 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.482518    5415 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:15.569450 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.569422    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qwnr5_81594aa6-a125-4f16-9c62-5b3b6f24c6fb/manager/0.log"
Oct 31 04:59:15.854616 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:15.854588    5415 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"]
Oct 31 04:59:15.859007 ip-10-0-139-164 kubenswrapper[5415]: W1031 04:59:15.858972    5415 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod244ce30f_9471_4bcc_a3ef_9e398eb40585.slice/crio-3dc31e9a0d67fea25b6f2c4fdd483719227909bb892c7e64a0c92f4923dc5f94 WatchSource:0}: Error finding container 3dc31e9a0d67fea25b6f2c4fdd483719227909bb892c7e64a0c92f4923dc5f94: Status 404 returned error can't find the container with id 3dc31e9a0d67fea25b6f2c4fdd483719227909bb892c7e64a0c92f4923dc5f94
Oct 31 04:59:16.116473 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:16.116368    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648" event={"ID":"244ce30f-9471-4bcc-a3ef-9e398eb40585","Type":"ContainerStarted","Data":"17b0296aee646aa14afe2341bf81c4bd1b43f7ff044ecb285ddf8d64f3d2c60f"}
Oct 31 04:59:16.116980 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:16.116513    5415 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648" event={"ID":"244ce30f-9471-4bcc-a3ef-9e398eb40585","Type":"ContainerStarted","Data":"3dc31e9a0d67fea25b6f2c4fdd483719227909bb892c7e64a0c92f4923dc5f94"}
Oct 31 04:59:16.116980 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:16.116540    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:16.182425 ip-10-0-139-164 kubenswrapper[5415]: E1031 04:59:16.182392    5415 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8452g" containerName="extractor"
Oct 31 04:59:16.227574 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:16.227498    5415 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648" podStartSLOduration=1.227480045 podStartE2EDuration="1.227480045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:59:16.224564772 +0000 UTC m=+6697.785044502" watchObservedRunningTime="2026-10-31 04:59:16.227480045 +0000 UTC m=+6697.787959774"
Oct 31 04:59:16.311644 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:16.311578    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nmdgv_6e19afe0-f05b-4294-8d01-8d04cbb5b833/manager/0.log"
Oct 31 04:59:16.424682 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:16.424586    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cmm9t_e94e222c-4de2-4046-a659-0c92420cc03e/manager/0.log"
Oct 31 04:59:16.581288 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:16.581257    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cmm9t_e94e222c-4de2-4046-a659-0c92420cc03e/manager/0.log"
Oct 31 04:59:16.789336 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:16.789249    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4m5hw_35e1e5d7-2240-4292-9f78-a9676f67e28a/server/0.log"
Oct 31 04:59:16.972187 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:16.972156    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4m5hw_35e1e5d7-2240-4292-9f78-a9676f67e28a/server/0.log"
Oct 31 04:59:18.776921 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:18.776861    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hpjht_6cb52f07-5218-4b28-896c-9b97e062ef44/manager/0.log"
Oct 31 04:59:18.955833 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:18.955793    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hpjht_6cb52f07-5218-4b28-896c-9b97e062ef44/manager/0.log"
Oct 31 04:59:18.967099 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:18.967044    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-bk6x5_9dcba361-a1d8-4e62-843c-be420bd3fea8/proxy/0.log"
Oct 31 04:59:19.120585 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:19.120553    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-bk6x5_9dcba361-a1d8-4e62-843c-be420bd3fea8/proxy/0.log"
Oct 31 04:59:22.132199 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:22.132172    5415 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-hs648"
Oct 31 04:59:22.138671 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:22.138648    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4kp58_da297bce-ca72-4dea-b02c-bcda6f401ebe/kube-state-metrics/0.log"
Oct 31 04:59:22.201708 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:22.201681    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qwnr5_81594aa6-a125-4f16-9c62-5b3b6f24c6fb/manager/0.log"
Oct 31 04:59:22.309304 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:22.309273    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qwnr5_81594aa6-a125-4f16-9c62-5b3b6f24c6fb/manager/0.log"
Oct 31 04:59:22.536858 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:22.535764    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4kp58_da297bce-ca72-4dea-b02c-bcda6f401ebe/kube-rbac-proxy-main/0.log"
Oct 31 04:59:22.938977 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:22.938906    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4kp58_da297bce-ca72-4dea-b02c-bcda6f401ebe/kube-rbac-proxy-self/0.log"
Oct 31 04:59:23.246189 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:23.246097    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cmm9t_e94e222c-4de2-4046-a659-0c92420cc03e/manager/0.log"
Oct 31 04:59:23.347744 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:23.347704    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7887dcbc55-2plp6_a4d955b0-1418-499f-9036-961443f1e130/metrics-server/0.log"
Oct 31 04:59:23.381056 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:23.381016    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cmm9t_e94e222c-4de2-4046-a659-0c92420cc03e/manager/0.log"
Oct 31 04:59:23.752488 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:23.752459    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-p2kxj_a18739ba-0914-4b52-bcda-4a5fafe22225/monitoring-plugin/0.log"
Oct 31 04:59:23.923452 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:23.923423    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4m5hw_35e1e5d7-2240-4292-9f78-a9676f67e28a/server/0.log"
Oct 31 04:59:24.129959 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:24.129928    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4m5hw_35e1e5d7-2240-4292-9f78-a9676f67e28a/server/0.log"
Oct 31 04:59:24.158951 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:24.158922    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2cjh2_177793bf-7bb1-4b0b-adb6-ba40483ae237/node-exporter/0.log"
Oct 31 04:59:24.568962 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:24.565835    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2cjh2_177793bf-7bb1-4b0b-adb6-ba40483ae237/kube-rbac-proxy/0.log"
Oct 31 04:59:24.966963 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:24.966926    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2cjh2_177793bf-7bb1-4b0b-adb6-ba40483ae237/init-textfile/0.log"
Oct 31 04:59:25.932870 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:25.932829    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hpjht_6cb52f07-5218-4b28-896c-9b97e062ef44/manager/0.log"
Oct 31 04:59:26.105426 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:26.105347    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-bk6x5_9dcba361-a1d8-4e62-843c-be420bd3fea8/proxy/0.log"
Oct 31 04:59:26.123765 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:26.123736    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hpjht_6cb52f07-5218-4b28-896c-9b97e062ef44/manager/0.log"
Oct 31 04:59:26.324178 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:26.324097    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-bk6x5_9dcba361-a1d8-4e62-843c-be420bd3fea8/proxy/0.log"
Oct 31 04:59:27.817239 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:27.817198    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-mbdfr_2941755b-c963-4514-a42a-c918a8283273/kube-rbac-proxy-main/0.log"
Oct 31 04:59:28.219990 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:28.219957    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-mbdfr_2941755b-c963-4514-a42a-c918a8283273/kube-rbac-proxy-self/0.log"
Oct 31 04:59:28.628276 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:28.626060    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-mbdfr_2941755b-c963-4514-a42a-c918a8283273/openshift-state-metrics/0.log"
Oct 31 04:59:29.044319 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:29.044241    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72e90f94-e4f3-4fd3-a667-de34caeb2d48/prometheus/0.log"
Oct 31 04:59:29.445414 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:29.445378    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72e90f94-e4f3-4fd3-a667-de34caeb2d48/config-reloader/0.log"
Oct 31 04:59:29.846489 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:29.846463    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72e90f94-e4f3-4fd3-a667-de34caeb2d48/thanos-sidecar/0.log"
Oct 31 04:59:30.249195 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:30.249092    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72e90f94-e4f3-4fd3-a667-de34caeb2d48/kube-rbac-proxy-web/0.log"
Oct 31 04:59:30.654147 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:30.654119    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72e90f94-e4f3-4fd3-a667-de34caeb2d48/kube-rbac-proxy/0.log"
Oct 31 04:59:31.056134 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:31.056058    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72e90f94-e4f3-4fd3-a667-de34caeb2d48/kube-rbac-proxy-thanos/0.log"
Oct 31 04:59:31.459939 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:31.459907    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72e90f94-e4f3-4fd3-a667-de34caeb2d48/init-config-reloader/0.log"
Oct 31 04:59:31.870795 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:31.870765    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-9pbqq_16799a99-285f-40d5-acbb-a8fbe0d7a2d9/prometheus-operator/0.log"
Oct 31 04:59:32.270231 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:32.270152    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-9pbqq_16799a99-285f-40d5-acbb-a8fbe0d7a2d9/kube-rbac-proxy/0.log"
Oct 31 04:59:32.681071 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:32.681037    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-wtgvk_0077be45-ce2b-46f0-8fe4-e881254f6386/prometheus-operator-admission-webhook/0.log"
Oct 31 04:59:34.470186 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:34.470157    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb6b5fd89-w7s8t_5ec9b273-54b7-49b5-bd15-09d2230e7c3c/kube-auth-proxy/0.log"
Oct 31 04:59:34.679057 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:34.679018    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-5smg5_1cfc6a64-edfa-4e51-83b5-f38f438f1877/istio-proxy/0.log"
Oct 31 04:59:34.780937 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:34.780849    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb6b5fd89-w7s8t_5ec9b273-54b7-49b5-bd15-09d2230e7c3c/kube-auth-proxy/0.log"
Oct 31 04:59:34.988293 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:34.988259    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-5smg5_1cfc6a64-edfa-4e51-83b5-f38f438f1877/istio-proxy/0.log"
Oct 31 04:59:35.303387 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:35.303359    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-bwzrc_c9d3bfd0-545c-4be0-92ae-30a7c68138ca/istio-proxy/0.log"
Oct 31 04:59:35.611008 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:35.610981    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-bwzrc_c9d3bfd0-545c-4be0-92ae-30a7c68138ca/istio-proxy/0.log"
Oct 31 04:59:36.551511 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:36.551479    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-wlt76_c0469dfd-14dc-458a-9b3e-d5b0ecc25000/istio-proxy/0.log"
Oct 31 04:59:36.864607 ip-10-0-139-164 kubenswrapper[5415]: I1031 04:59:36.864568    5415 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-wlt76_c0469dfd-14dc-458a-9b3e-d5b0ecc25000/istio-proxy/0.log"